Author |
Topic: Dynamic loading (Read 26792 times) |
|
Laurent Itti
YaBB Moderator
YaBB God
iLab rocks!
Gender: 
Posts: 550
|
 |
Dynamic loading
« on: 04/18/03 at 17:27:02 » |
 Quote  Modify
|
Hey all:
the code is getting increasingly slow to compile as more functionality becomes available as run-time plugins. This is not surprising, since now even the smallest program using, say, StdBrain, will link against a large collection of objects, including Beowulf objects and such. This is not a bug but a real feature, as this simple little program will indeed inherit the capability of running its channels onto a Beowulf.
Because CORE_OBJS in the Makefile is getting increasingly long, first, this makes linking a given file slower, and, second, it makes the likelihood that we will have to re-link everything increasingly likely (e.g., if I modify Beowulf.C, all CORE_OBJ programs will need to be re-linked).
One way out, it seems, would be to have dynamic loading, but I don't know anything about that. The idea would be to keep the pieces separate in a collection of dynamic libraries that would be loaded at run-time rather than statically linked.
does this sound appealing to anybody? does anyone have any experience with that kind of stuff?
thanks!
-- laurent
|
|
 Logged |
|
|
|
Nathan Mundhenk
YaBB Full Member
HA!
  
Gender:
Posts: 140
|
|
Re: Dynamic loading
« Reply #1 on: 04/18/03 at 20:53:10 » |
Quote Modify
|
This sounds cool. I was noticing that compile is starting to get a bit long, even for me. However, I to must admit that I know nothing about this. I'll poke around if no one knows anything about this... see if I can get some info. I think this also makes another argument for splitting the code. Since I don't need the latest version of brain for many of my things, I really don't need to compile it (or even have the latest version). It could be cool for me if I just do a total compile once a week or so and just compile my little bits more often, not everytime I do a CVS update, or is that perhaps problematic. Not sure at NE rate.
It could be interesting also if we broke the code in to packages. Who ever is hacking at that package can make changes here and there and can release a final version when there done. From that everyone else who just uses the package would update. That way I don't have to recompile several things everyday. I would just recompile it when the package (that I'm not working on) reaches a release phase.
It might also be nice to have a central RPM server of some type. It would keep the latest compiled version of the code. Instead of the normal CVS update maybe we could use an RPM update. This way we would at least skip the compile to update. When you want to commit, you just do it in the old fashoned CVS manner. The trick would be to include the source and binaries in the RPM in a manner that is pleasing to all.
NE ways, just ideas. I suspect there is 1,000,000,000 ways to approach this problem.
 
|
|
Logged |
T. Nathan Mundhenk
http://www.nerd-cam.com
http://www.cool-ai.com
|
|
|
Laurent Itti
YaBB Moderator
YaBB God
iLab rocks!
Gender:
Posts: 550
|
|
Re: Dynamic loading
« Reply #2 on: 04/20/03 at 12:52:20 » |
Quote Modify
|
yep, sounds good. Looking around for dlopen and C++, it looks like dynamic loading of C++ classes at run time is not trivial...
yes, you make a good point about splitting the code. The only danger there is that if some of us get into the habit of not always recompiling everything, then we'll see an increasing number of commits that break code that you have not looked at... it's a bit the same problem with packages: if I knew what the final interface to my package will be, then yes I could be developping it in my corner and not bother the others with my updates; but typically while we develop a package we also substantially change its interface, so anybody using it will be affected.
one way of segmenting that better, though, could be to have more make targets, maybe one for each developer, that you would put in your Makefile.local. Then, while coding, you just build your own programs, and it's only when you are ready to commit that you do a make all/make test to be sure that nothing is broken.
I like the idea of the RPM server too. Has anybody experience with writing RPM specs? Since we already do a CVS checkout o the entire tree every night to generate the online doc, we could run that in the same script.
thanks for the comments!
-- laurent
|
|
Logged |
|
|
|
Juliet
YaBB Newbie
Ciao!
Gender: 
Posts: 5
|
|
Re: Dynamic loading
« Reply #3 on: 04/22/03 at 01:13:43 » |
Quote Modify
|
Greetings,
I dont know much about the dynamic linking either  but did some research:
This concept is known as shared libraries. If you want small binaries, less compilation time and do the linking during the run time this is the way to go, but consider the fact that it will increase your run time.
Please be aware that your code is not suppossed to have any static objects if you want to use dynamic linking.
Here is a simple example I've found on how to create a shared library:
/* file shrobj.c */
const char *myfunc()
{
return "Hello!";
}
/* file hello.c */
#include <stdio.h>
extern const char *myfunc();
main()
{
printf ("%s\n", myfunc());
return 0:
}
$ gcc -fpic -c shrobj.c
$ gcc -shared -o libshared.so shrobj.o
$ gcc hello.c libshared.so
$ ./a.out
Hello!
Name of shared libraries end with .so ( .a for static libraries)
If your shared library has a name as "libfoo.so", then in your makefile you should use -lfoo. By default the system will look for the dynamic library "libfoo.so" and in the absence of that one it will search for the static library "libfoo.a". You can use -Bdynamic to force to find the shared library.
Use man pages for -ld, it has some useful information.
Sorry for not providing you detailed info since I am very busy with work. I must admit that this thing got me confused, need to experiment it  .
Following link also has detailed explanation about creation, installation and usage of shared libraries:
http://tldp.org/HOWTO/Program-Library-HOWTO/shared-libraries.html
The other subject about using the packages or targets: I dont know how your file structure is, but another approach you can take is simply separating your source code into different directories based on their functionality.
For ex. shared, brain, ui, monitor, hardware
Each directory can have it's own src, inc, bin and lib directories. By using separate makefiles you can generate a library for each functional area and store it under the related lib directory. Then when you execute your code you can include the "necessary" libraries to be linked into your executable. Dont know if this is what you mean w/ using packages 
hope this helps...
j
|
|
Logged |
|
|
|
Laurent Itti
YaBB Moderator
YaBB God
iLab rocks!
Gender:
Posts: 550
|
|
Re: Dynamic loading
« Reply #4 on: 04/22/03 at 10:03:48 » |
Quote Modify
|
Hi Juliet:
this looks good, except that if I am not mistaken, each time I change anything in shrobj.c, I will have to re-link everything?
I think that with dlopen() we can avoid that, at the cost of having to explicitly request each shared function we need. I got that so far, and looks like with C++ things get tricky:
http://www.tldp.org/HOWTO/mini/C++-dlopen/
best,
-- laurent
|
|
Logged |
|
|
|
Nathan Mundhenk
YaBB Full Member
HA!
Gender:
Posts: 140
|
|
Re: Dynamic loading
« Reply #5 on: 04/22/03 at 15:58:16 » |
Quote Modify
|
Looking at the dlopen syntax it seems to me that it could be a pain to rewrite some stuff in this manner. I think if this becomes our solution we should figure out how things can be re-writen and where that would happen. We should also consider testing performance. in short this could be quite a change to the code if dlopen is used, but I must admit, it could be best solution.
|
|
Logged |
T. Nathan Mundhenk
http://www.nerd-cam.com
http://www.cool-ai.com
|
|
|
Forum Admin
YaBB Administrator
iLab rocks!
Gender:
Posts: 6
|
|
Re: Dynamic loading
« Reply #6 on: 04/22/03 at 18:49:29 » |
Quote Modify
|
yeah, it does look like a major headache, especially for C++. I think a 64-CPU compilation server with shared memory is what we need instead 
best,
-- laurent
|
|
Logged |
|
|
|
Juliet
YaBB Newbie
Ciao!
Gender:
Posts: 5
|
|
Re: Dynamic loading
« Reply #7 on: 04/23/03 at 00:12:51 » |
Quote Modify
|
on 04/22/03 at 10:03:48, Laurent Itti wrote:
this looks good, except that if I am not mistaken, each time I change anything in shrobj.c, I will have to re-link everything?
|
|
You dont have to relink everything. Dynamic linking should allow an error to be fixed w/o relinking.
One thing I forgot to mention was that you also need to set the environment variable LD_LIBRARY_PATH to point to the directory where the shared libraries located. Probably you already know this...
yes C++ seems like trickier than C, do you really have to use this dlopen()?
j
|
|
Logged |
|
|
|
Dirk Bernhardt-Walther
YaBB Full Member
Posts: 155
|
|
Re: Dynamic loading
« Reply #8 on: 04/23/03 at 05:27:33 » |
Quote Modify
|
Do we really have to do dynamic linking? The code is slowly recovering from our last system-wide improvements, and I wouldn't mind having it stable for a week or two to focus on other things than fixing broken code...
Dirk
|
|
Logged |
|
|
|
Laurent Itti
YaBB Moderator
YaBB God
iLab rocks!
Gender:
Posts: 550
|
|
Re: Dynamic loading
« Reply #9 on: 04/23/03 at 10:35:52 » |
Quote Modify
|
Hi Juliet;
oh, I see, so probably the
gcc hello.c libshared.so
is not necessary unless you have changed some of your interface in libshared? I am wondering how you would make that into a Makefile?
libshared.so: libshared.h libshared.c
compile it
hello: hello.c libshared.h
compile it
so that only when libshared.h (the interface definition) is changed would re-linking be done?
but overall, I agree with Dirk too :-P just it's taking upwards of 7:20 minutes to do an "mk all" after an "mk clean" which is kind of long... One simple thing I will look into is reducing the number of our little test programs; e.g., for the pvision* stuff, we could only keep the best version and phase out the older ones, which are seldom used anyways.
best,
-- laurent
|
|
Logged |
|
|
|
Dirk Bernhardt-Walther
YaBB Full Member
Posts: 155
|
|
Test programs
« Reply #10 on: 04/23/03 at 10:45:46 » |
Quote Modify
|
Let's make a separate make target for all the test programs - as 'test' is already taken, perhaps let's use something like 'tprogs' (for test programs) or so - then we'd still have to re-compile everything at 'make all', but not necessarily at 'make'.
Dirk
|
|
Logged |
|
|
|
Juliet
YaBB Newbie
Ciao!
Gender:
Posts: 5
|
|
Re: Dynamic loading
« Reply #11 on: 04/24/03 at 02:03:06 » |
Quote Modify
|
on 04/23/03 at 10:35:52, Laurent Itti wrote:Hi Juliet;
oh, I see, so probably the
gcc hello.c libshared.so
is not necessary unless you have changed some of your interface in libshared?
|
|
Hi, sorry for the late replies 
Yes, exactly. So when you change shrobj.c then you need to redo the first 2 lines:
gcc -fpic ...
gcc -shared ...
however when you change the header file which is included in the target file, then you must relink as well. But this is a partial linking where linker only puts the symbol references into the program. The real object file insertion is done by the dynamic loader during the run time. Also if there is a second program linked with the same shared library, it can use the same copy of the shared library, thus saving a lot of memory.
Makefile? Himmm, isnt it better to have 2 Makefiles one to create the library and one for the executable? If I get a chance I might write a little sample tomorrow.
BTW, this dlopen() is used to explicitly state which functions to use from the shared library instead of loading the whole thing. It requires quite a bit change to the code unfortunatelly.
Long compilation time is a common problem for this size code, how about generating multiple libraries as I mentioned before? But that might also require a small (!)amount of change which might make life more fun...
regards
j
|
|
Logged |
|
|
|
Laurent Itti
YaBB Moderator
YaBB God
iLab rocks!
Gender:
Posts: 550
|
|
Re: Dynamic loading
« Reply #12 on: 04/24/03 at 10:01:35 » |
Quote Modify
|
great, thanks much for the clarifications!
I have a few more questions: first, is the linking against the .so really faster than a static linking, in actual practice? It should be since probably importing the symbols can be done just by looking at a small portion at the .so file rather than loading it all up as in static linking.
second, it seems that your approach would actually require no change (or very little) in the code proper, but just some re-organization in the Makefile. Would that be correct even for C++? How about the extreme case where we make a .so for each of our classes (e.g., VisualCortex.so, Brain.so, WTA.so, etc), it seems that we could then just do a fast link of each executable against the collection of .so files it needs and we would be set? More reasonably, we could have a collection of shared libs for each theme (e.g., Image.so with all the Image_*.C stuff, Beowulf.so with all the Beowulf/TCPmessage stuff, etc), basically along the lines of the IMAGE_OBJS, etc we currently have in the makefile
third, we would need to verify that indeed it is okay to not relink if a .so gets recompiled but there was no change in the interface (.H), so that we would avoid re-linking everybody as long as only implementations of libs are being changed but not their interface. Now I am still unclear on how we would do that in a makefile:
- we want the executable to NOT depend on the .so, otherwise we will have to re-link everytime the .so is changed, which would be similar to what we have now (though re-linking may be faster)
- so we could make the executable depend on the corresponding .H files instead
- but we would need to enforce that whenever a .H is modified, the .so is compiled first, and only thereafter are all the executables that use that .so relinked
Otherwise, I like Dirk's idea of a separate target for test programs, and maybe that would be a good first step.
Thanks!
-- laurent
|
|
Logged |
|
|
|
Juliet
YaBB Newbie
Ciao!
Gender:
Posts: 5
|
|
Re: Dynamic loading
« Reply #13 on: 04/25/03 at 01:26:38 » |
Quote Modify
|
Hawdy!
I have written couple makefiles and tested them. have slightly changed the earlier source codes to create C++ versions, here they are:
Precious.h:
class Precious
{
public:
Precious() {;}
const char* myfunction();
};
Precious.c:
#include "Precious.h"
const char* Precious::myfunction()
{
return "Message from Precious, proud to be shared!";
}
Driver.c:
#include <iostream.h>
#include "Precious.h"
int main()
{
Precious p;
cout<<p.myfunction()<<'\n';
return 0;
}
We have 2 makefiles, first one to compile and generate the library and second one for the target:
--MakeLib--
# Sample Makefile to build object files and shared libraries
## Definitions ##
# compiler specs
CPP= /usr/local/bin/g++
CPPFLAGS= -g -Wall
# objects
OBJECTS= Precious.o
# libraries
LSHARED= libshared.so
# source files
SRC= Precious.c
## Rules ##
ALL: $(OBJECTS) $(LSHARED)
# dependencies (clearmake implicitly knows about dependencies)
Precious.o: Precious.h
$(OBJECTS): $(SRC)
$(CPP) $(CPPFLAGS) -fpic -c $(SRC)
$(LSHARED): $(OBJECTS)
$(CPP) -shared -o $(LSHARED) $(OBJECTS)
clean:
rm -rf *o
--MakeTarget--
# Sample Makefile to build target file using libraries
## Definitions ##
# compiler specs
CPP= /usr/local/bin/g++
CPPFLAGS= -g -Wall
# source files
SRC= Driver
# libraries (by default OS will search dynamic first and static next)
LIBS= -lshared
# target file
TARGET= runme
## Rules ##
ALL: $(TARGET)
# dependencies (clearmake implicitly knows about dependencies)
$(SRC).o: Precious.h
$(SRC).o: $(SRC).c
$(CPP) $(CPPFLAGS) -fpic -c $(SRC).c
$(TARGET): $(SRC).o
$(CPP) $(SRC).o -o $(TARGET) $(LIBS)
clean:
rm -rf *o
Now, here when you make a change to Precious.c than you only have to run MakeLib. But if you change the dependency file--Precious.h--than you have to run both makefiles. I made sure that the header file is in the dependency list of both makefiles so that they will execute if the version of the header file has been changed.
Can you get away w/o changing the source codes?
I hope so
but it's hard to tell since the examples I have here are way too simple.
I think best efficiency is gonna come from modulization, in other words grouping the files and generating libraries for each functional area and then linking those libs for different targets. probably you guys should give a small scale try to this and see what it's gonna do, otherwise you can try other methods.
good luck 
juliet
|
|
Logged |
|
|
|
Dirk Bernhardt-Walther
YaBB Full Member
Posts: 155
|
|
Re: Dynamic loading
« Reply #14 on: 04/25/03 at 07:28:25 » |
Quote Modify
|
Quote: Otherwise, I like Dirk's idea of a separate target for test programs, and maybe that would be a good first step.
|
|
Done.
Dirk
|
|
Logged |
|
|
|
|
Home
Help
Search
Members
Login
Register
Reply
Notify of replies
Send Topic
Print