Hidden diversion - "Nested Tcl_Obj deletion management support".

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Hidden diversion - "Nested Tcl_Obj deletion management support".

tcl-core mailing list
Hello.

This code breaks my program.

tclObj.c

/*
 * Nested Tcl_Obj deletion management support
 *
 * All context references used in the object freeing code are pointers to this
 * structure; every thread will have its own structure instance. The purpose
 * of this structure is to allow deeply nested collections of Tcl_Objs to be
 * freed without taking a vast depth of C stack (which could cause all sorts
 * of breakage.)
 */

typedef struct PendingObjData {
    int deletionCount;          /* Count of the number of invokations of
                                 * TclFreeObj() are on the stack (at least
                                 * conceptually; many are actually expanded
                                 * macros). */
    Tcl_Obj *deletionStack;     /* Stack of objects that have had TclFreeObj()
                                 * invoked upon them but which can't be
                                 * deleted yet because they are in a nested
                                 * invokation of TclFreeObj(). By postponing
                                 * this way, we limit the maximum overall C
                                 * stack depth when deleting a complex object.
                                 * The down-side is that we alter the overall
                                 * behaviour by altering the order in which
                                 * objects are deleted, and we change the
                                 * order in which the string rep and the
                                 * internal rep of an object are deleted. Note
                                 * that code which assumes the previous
                                 * behaviour in either of these respects is
                                 * unsafe anyway; it was never documented as
                                 * to exactly what would happen in these
                                 * cases, and the overall contract of a
                                 * user-level Tcl_DecrRefCount() is still
                                 * preserved (assuming that a particular T_DRC
                                 * would delete an object is not very
                                 * safe). */
} PendingObjData;

...

void
TclFreeObj(
    register Tcl_Obj *objPtr)   /* The object to be freed. */
{
    /*
     * Invalidate the string rep first so we can use the bytes value for our
     * pointer chain, and signal an obj deletion (as opposed to shimmering)
     * with 'length == -1'.
     */

    TclInvalidateStringRep(objPtr);
    objPtr->length = -1;

    if (!objPtr->typePtr || !objPtr->typePtr->freeIntRepProc) {
        /*
         * objPtr can be freed safely, as it will not attempt to free any
         * other objects: it will not cause recursive calls to this function.
         */

        TCL_DTRACE_OBJ_FREE(objPtr);
        TclFreeObjStorage(objPtr);
        TclIncrObjsFreed();
    } else {
        /*
         * This macro declares a variable, so must come here...
         */

        ObjInitDeletionContext(context);

        if (ObjDeletePending(context)) {
            PushObjToDelete(context, objPtr);
        } else {
            /*
             * Note that the contents of the while loop assume that the string
             * rep has already been freed and we don't want to do anything
             * fancy with adding to the queue inside ourselves. Must take care
             * to unstack the object first since freeing the internal rep can
             * add further objects to the stack. The code assumes that it is
             * the first thing in a block; all current usages in the core
             * satisfy this.
             */

            TCL_DTRACE_OBJ_FREE(objPtr);
            ObjDeletionLock(context);
            objPtr->typePtr->freeIntRepProc(objPtr);
            ObjDeletionUnlock(context);

            TclFreeObjStorage(objPtr);
            TclIncrObjsFreed();
            ObjDeletionLock(context);
            while (ObjOnStack(context)) {
                Tcl_Obj *objToFree;

                PopObjToDelete(context, objToFree);
                TCL_DTRACE_OBJ_FREE(objToFree);
                if ((objToFree->typePtr != NULL)
                        && (objToFree->typePtr->freeIntRepProc != NULL)) {
                    objToFree->typePtr->freeIntRepProc(objToFree);
                }
                TclFreeObjStorage(objToFree);
                TclIncrObjsFreed();
            }
            ObjDeletionUnlock(context);
        }
    }
    ...
}


I have rewritten it.

void
TclFreeObj(
    register Tcl_Obj *objPtr)   /* The object to be freed. */
{

    TCL_DTRACE_OBJ_FREE(objPtr);
    if (objPtr->typePtr && objPtr->typePtr->freeIntRepProc)
            objPtr->typePtr->freeIntRepProc(objPtr);
    TclInvalidateStringRep(objPtr);
    objPtr->length = -1;
    TclFreeObjStorage(objPtr);
    TclIncrObjsFreed();
    ...
}

And everything became correct to work.

I ask to delete "Nested Tcl_Obj deletion management support". If this code is
necessary to someone then to make it a compilation option.

/foo

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Tcl-Core mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/tcl-core
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Hidden diversion - "Nested Tcl_Obj deletion management support".

Donald Porter

> On Jun 25, 2017, at 2:45 PM, foo via Tcl-Core <[hidden email]> wrote:
> This code breaks my program.

The code you quoted does not appear in any current release
of Tcl.

Please check whether your program is happier with our current
release, Tcl 8.6.6.  If absolutely necessary, you might also try
Tcl 8.5.19.  The last time Tcl looked somewhat like what you
quoted appears to be 8.5.17 or 8.6.3 in 2014.

If your program was working with some much older version
of Tcl before the pending delete scheme was put in place,
that was indeed a long time ago.  All releases of Tcl since
around 2004 have included that.

DGP


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Tcl-Core mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/tcl-core
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Hidden diversion - "Nested Tcl_Obj deletion management support".

Donal K. Fellows-2
In reply to this post by tcl-core mailing list
On 25/06/2017 19:45, foo via Tcl-Core wrote:
> This code breaks my program.

Might I inquire why your program is such that postponing the delete
breaks things? It's never been particularly safe to rely on an object
contained in another object being deleted at a specific time, as Tcl
really doesn't try to avoid sharing the references, and the use of
thread-specific data (or a static in a non-threaded build) to manage the
free list should be invisible to code outside of Tcl's implementation.

> It uses the Tcl_ObjType mechanism to define new types directly from
> tcl source code. It allows to perform operations in case of automatic
> destruction of variables of this type.

Ah. That's not supported partially because it's got too many problems
with reentrant code, and also because we don't support fragile
references in the first place (Tcl mutates types a bit more readily than
you seem to hope in some usage patterns). I wonder whether it would be
possible to treat the loss of the last reference as being a signal to
put the object on a queue for deletion that you then process
periodically. After all, that's a scheme that is used by some GC
systems, and allows your on-deletion callbacks to be processed at a time
when the interpreter state is such that it is safe to call the API.

Generally speaking, the sort of garbage collection done by Tcl isn't
enough for you when you're doing objects of the kind you're probably
talking about, as the mutability means that you can create reference
loops, and refcounts cannot collect those. Instead, you need some more
complex full GC. By comparison, TclOO doesn't have this problem
precisely because it just uses explicit deletion; however, where there
is a well-defined ownership relation — equivalent to the UML Composition
relation — it's pretty trivial to automatically tie the lifetime of the
contained object to that of the container by putting the containee's
handle command inside the container's namespace.

For the record, I'd like TclOO to be able to support the sort of GC
you're talking about when objects are built in “nameless” mode with the
'new' method; since by definition existing scripts would need to keep a
hold of them to use them, that would not break anything. (Objects made
with 'create' would continue to require explicit deletion though; since
that includes all classes, I think that's a good idea. Also, using
[rename] on a “nameless” object would have to take it out of the GC
system.) It is quite a technical challenge to make that work though, so
it's been rather on my back-burner, and it requires some tricky changes
in the guts of Tcl (as the commands so created would have to not shimmer
to the standard command Tcl_ObjType to work).

Donal.

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Tcl-Core mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/tcl-core

donal_k_fellows.vcf (241 bytes) Download Attachment
dah
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Hidden diversion - "Nested Tcl_Obj deletion management support".

dah
On 6/27/2017 3:54 PM, Donal K. Fellows wrote:

> I wonder whether it would be possible to treat the loss of the last
> reference as being a signal to put the object on a queue for deletion
> that you then process periodically.

Why doesn't Tcl itself just do this?  Instead of burdening the user with
manual destruction and resource leaks? Queued periodic deletion would be
a very welcome next step. Sure there is TIMTOWDI to destroy them but all
are a burden and DIY. For those that love DIY and fixing their own bugs
that is great. But not for the rest of us that don't wish to be burdened
by such tediums.   Out of all the languages I use most often, Tcl is the
only one where I have to constantly concern myself with resource leaks
when I'm doing anything with TclOO. Most of the time, I just want to
create an object based off some input, do something with it temporarily
and then be done with it. And if need it longer, I always store the
objects in a global registry.  Things get a little more complicated when
you're working in layers of abstraction. Believe me, I slap myself daily
for using TclOO for anything. I should've stuck with dicts and
disjointed procs to operate on them.

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Tcl-Core mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/tcl-core
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Hidden diversion - "Nested Tcl_Obj deletion management support".

tcl-core mailing list
In reply to this post by Donal K. Fellows-2



Среда, 28 июня 2017, 2:54 +07:00 от "Donal K. Fellows" <[hidden email]>:

On 25/06/2017 19:45, foo via Tcl-Core wrote:
> This code breaks my program.

Might I inquire why your program is such that postponing the delete
breaks things?
Because freeIntRepProc() make call of destructor of user type. Destructor make call
of Tcl_GetString() ... and "bus error" or "segmentation fault".  You guess why?

It's never been particularly safe to rely on an object
contained in another object being deleted at a specific time, as Tcl
really doesn't try to avoid sharing the references, and the use of
thread-specific data (or a static in a non-threaded build) to manage the
free list should be invisible to code outside of Tcl's implementation.

> It uses the Tcl_ObjType mechanism to define new types directly from
> tcl source code. It allows to perform operations in case of automatic
> destruction of variables of this type.

Ah. That's not supported partially because it's got too many problems
with reentrant code, and also because we don't support fragile
references in the first place (Tcl mutates types a bit more readily than
you seem to hope in some usage patterns).

Variability of internal representation can be controlled by the programmer. Don't
substitute the wrong types as arguments of function. This rather obvious and
simple rule.

I wonder whether it would be
possible to treat the loss of the last reference as being a signal to
put the object on a queue for deletion that you then process
periodically. After all, that's a scheme that is used by some GC
systems, and allows your on-deletion callbacks to be processed at a time
when the interpreter state is such that it is safe to call the API.
At you everything is very difficult :) Not always it is necessary to think out the new
and difficult solution, especially if it already exists and works (as I see). It isn't
necessary to postpone a call of destructors and release of memory when it can be
done simply and now.

Generally speaking, the sort of garbage collection done by Tcl isn't
enough for you when you're doing objects of the kind you're probably
talking about, as the mutability means that you can create reference
loops, and refcounts cannot collect those.
???

Instead, you need some more
complex full GC.

Oh, No. I like small and simple solutions.

By comparison, TclOO doesn't have this problem
precisely because it just uses explicit deletion; however, where there
is a well-defined ownership relation — equivalent to the UML Composition
relation — it's pretty trivial to automatically tie the lifetime of the
contained object to that of the container by putting the containee's
handle command inside the container's namespace.

For the record, I'd like TclOO to be able to support the sort of GC
you're talking about when objects are built in “nameless” mode with the
'new' method; since by definition existing scripts would need to keep a
hold of them to use them, that would not break anything. (Objects made
with 'create' would continue to require explicit deletion though; since
that includes all classes, I think that's a good idea. Also, using
[rename] on a “nameless” object would have to take it out of the GC
system.) It is quite a technical challenge to make that work though, so
it's been rather on my back-burner, and it requires some tricky changes
in the guts of Tcl (as the commands so created would have to not shimmer
to the standard command Tcl_ObjType to work).

Donal.
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Tcl-Core mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/tcl-core


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Tcl-Core mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/tcl-core
dah
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Hidden diversion - "Nested Tcl_Obj deletion management support".

dah
> At you everything is very difficult :) Not always it is necessary to
> think out the new
> and difficult solution, especially if it already exists and works (as
> I see). It isn't
> necessary to postpone a call of destructors and release of memory when
> it can be
> done simply and now.

I doubt it was designed with a clean way to do RAII (which is
essentially what we're asking for). DKF seems to like keeping things
nice and clean wherever possible. So it may be difficult to impossible
to do cleanly given overall design, and then there's difficulty in
acceptance -- there must be a TIP, a vote, etc. Nothing is easy with Tcl.

Tcl'ers love to craft things themselves and you'll find dozens of
homegrown implementations of auto-cleanup for these types of things. I
can think of several myself, and you can retrofit all your code to use
them. Yeah, that sucks, but it is the Tcl way. For me, I just want THE
way.  That said, If you get something to fly (if you're able to hack Tcl
internals to do it), I would love to see it.


On 6/28/2017 1:26 PM, Александр Митрохин via Tcl-Core wrote:
> For the record, I'd like TclOO to be able to support the sort of GC
> you're talking about when objects are built in “nameless” mode with the
> 'new' method; since by definition existing scripts would need to keep a
> hold of them to use them, that would not break anything.
>
> Donal.

That will break code though, since objects aren't POD. You can launch an
object into the void and have it do work (with side effects) and not
ever care about talking to it again. I'll bet there's plenty of
instances of this, so unless TclOO made it clear in docs that you must
hold a reference to guarantee its life.

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Tcl-Core mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/tcl-core
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Hidden diversion - "Nested Tcl_Obj deletion management support".

Donal K. Fellows-2
On 28/06/2017 20:51, dah wrote:
> I doubt it was designed with a clean way to do RAII (which is
> essentially what we're asking for).

Well, it's not too hard to do things like that, but it requires either
tying to the lifespan of a local variable (via an unset trace) or
switching to a user-generated-block approach. That latter one is done
with something like this:

   oo::class create Context {
       method as {varname body in} {
           upvar 1 $varname v
           set v [self]
           try {
               uplevel 1 $body
           } finally {
               unset -nocomplain v
               catch {my destroy}
           }
       }
   }

That can be then used like this:

   # Add to a simple class as a mixin; subclass would also work
   oo::class create FileExample {
       mixin Context
       variable f
       constructor {filename} {
           set f [open $filename]
       }
       destructor {
           close $f
       }
       method gets {} {
           gets $f
       }
   }

   # Use it with a block
   [FileExample /tmp/foo.txt] as channel in {
       puts [$channel gets]
       puts [$channel gets]
   }

Given the above, which I assert is pretty Tcl-ish, I think we don't miss
RAII very much at all! RAII solves a problem we can easily choose to not
have in the first place.

Donal.

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Tcl-Core mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/tcl-core

donal_k_fellows.vcf (241 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Hidden diversion - "Nested Tcl_Obj deletion management support".

Donal K. Fellows-2
On 29/06/2017 20:20, Donal K. Fellows wrote:

>   oo::class create Context {
>       method as {varname body in} {
>           upvar 1 $varname v
>           set v [self]
>           try {
>               uplevel 1 $body
>           } finally {
>               unset -nocomplain v
>               catch {my destroy}
>           }
>       }
>   }
It has been pointed out to me that I shouldn't write syntactic sugar
when very tired. The argument list to that method therefore should be:
{varname in body}

Donal.

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Tcl-Core mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/tcl-core

donal_k_fellows.vcf (241 bytes) Download Attachment
dah
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Hidden diversion - "Nested Tcl_Obj deletion management support".

dah
In reply to this post by Donal K. Fellows-2
Thanks, that is good stuff, and I believe those sorts of useful
constructs should be included in the core distribution so we get THE way
to deal with a common problem that is _peer reviewed_ and not rolling
our own buggy implementations we wrote while half-asleep or beat from a
hard day (Happens to everyone).

Acquisition is typically easy and done once, but releasing isn't so much
(exceptions, multiple return points, you simply forgot to release,
you/someone else adds a new early return and forgot all about that
object).  I've seen programs eat several hundred megs per hour simply
because someone forgot to destroy somewhere. I expect to have to deal
with these details in lower level languages, not the higher levels.

To the OP guy. I would suggest looking at how tdom does it. Anytime you
treat an object as a command, it'll get morphed into a cmdNameType
object. Tcl will obliterate your object when it does its conversion, so
your attempt to be clever shiny new object type effectively does squat
AFAICT. Tdom has a hacky workaround to handle this.

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Tcl-Core mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/tcl-core
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Hidden diversion - "Nested Tcl_Obj deletion management support".

tcl-core mailing list
In reply to this post by Donal K. Fellows-2



Пятница, 30 июня 2017, 2:21 +07:00 от "Donal K. Fellows" <[hidden email]>:

On 28/06/2017 20:51, dah wrote:
> I doubt it was designed with a clean way to do RAII (which is
> essentially what we're asking for).

Well, it's not too hard to do things like that, but it requires either
tying to the lifespan of a local variable (via an unset trace) or
switching to a user-generated-block approach. That latter one is done
with something like this:

   oo::class create Context {
       method as {varname body in} {
           upvar 1 $varname v
           set v [self]
           try {
               uplevel 1 $body
           } finally {
               unset -nocomplain v
               catch {my destroy}
           }
       }
   }

That can be then used like this:

   # Add to a simple class as a mixin; subclass would also work
   oo::class create FileExample {
       mixin Context
       variable f
       constructor {filename} {
           set f [open $filename]
       }
       destructor {
           close $f
       }
       method gets {} {
           gets $f
       }
   }

   # Use it with a block
   [FileExample /tmp/foo.txt] as channel in {
       puts [$channel gets]
       puts [$channel gets]
   }

Given the above, which I assert is pretty Tcl-ish, I think we don't miss
RAII very much at all! RAII solves a problem we can easily choose to not
have in the first place.
Ok. And why not to make a traditional code steady against losses of resources?

My example:
do_something [constructor_with_allocation_of_resources ...] ...

Now it will lead to loss of resources, because the destructor won't be called. Let's try to correct it.

I have written very little tested prototype. Tomorrow I will try to make the description of the
internals of a package and to lay out the source file that it was possible to review approach.

Donal.
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Tcl-Core mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/tcl-core


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Tcl-Core mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/tcl-core
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Hidden diversion - "Nested Tcl_Obj deletion management support".

tcl-core mailing list

Вторник, 4 июля 2017, 1:21 +07:00 от Александр Митрохин via Tcl-Core <[hidden email]>:


:) Failure... Everything falls on the simplest things. And it seems to me that
tclsh is guilty of it more, than I.

And nevertheless.

A set of functions for definition and interaction with the user types (Types
are similar to struct {...} of language C):

typedef <typename> <ctor_args> <ctor_body> <dtor_body> <clone_body>
        Sets the name of type, his constructor, a destructor and the commands
        executed at cloning of data. Inside <ctor_body>, <dtor_body> and
        <clone_body>, an object of type is available as $_.

<typename> ...
        Creation of an object of the specified type.

tset $x <key> <value>
        Operation of change of fields of an object.

tget $x <key>
        Operation of data acquisition, stored in fields of an object.

clone $x
        Operation of cloning of an object.

keys $x
        Obtaining list of fields of an object.

typeof $x
        Obtaining name of type.


Example:

        typedef file_t {filename} {
                        tset $_ filename $filename
                        tset $_ fp [open $filename]
                } {
                        fclose $_
                } {
                        tset $_ fp [open [tget $_ filename]]
                }

        proc freadln {f} {
                return [gets [tget $f fp]]
        }
        proc fclose {f} {
                close [tget $f fp]
        }

        proc main {} {
                set f [file_t "foo.txt"]
                puts "first line: [freadln $f]"
                close $f
        }

        main

:) It is better not to start an example not to be upset.

Internal representation of an object consists of two parts. Assignment
operation "set a $b" increases the counter of links to external
structure, and both are variables will use one internal representation.
We can have 10 variables, they will refer on one external structure
with refCount 10 and the inner structure will have refCount 1. Changes
of data through any link, will be visible to remaining.

Operation of cloning is necessary. For this purpose it is enough to create
new external structure, which will refer to old internal structure which
refCount at the same time we increase. This operation also very simple.

Data reading doesn't pay attention to the counter of references
of internal structure. Creation of the personal copy of internal structure
happens on write.

Easy and convenient way of data presentation. :)

 http://tcl.uk.to/tcltypedef-prototype.tar.gz 

/foo




------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Tcl-Core mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/tcl-core
Loading...