[Netkit.users] Building a Custom Netkit Kernel
Costa >>
rui.rcosta at gmail.com
Tue Sep 20 17:21:02 CEST 2011
Dear Massimo,
here are the outputs you requested:
>
>user-netkit at HPProliant:~/Desktop/netkit/netkit$ vstart pc1 --print
>
>============= Starting virtual machine "pc1" =============
> Kernel:
/home/user-netkit/Desktop/netkit/netkit/kernel/torvalds-linux-v2.6.28-0/linux
> Memory: 256 MB
> Model fs: /home/user-netkit/Desktop/netkit/netkit/fs/netkit-fs
> Filesystem: /home/user-netkit/Desktop/netkit/netkit/pc1.disk (new)
> Hostfs at: /home/user-netkit
>
>Not running ==> xterm -e
/home/user-netkit/Desktop/netkit/netkit/kernel/torvalds-linux-v2.6.28-0/linux
name=pc1 title=pc1 umid=pc1 >mem=260M
ubd0=/home/user-netkit/Desktop/netkit/netkit/pc1.disk,/home/user-netkit/Desktop/netkit/netkit/fs/netkit-fs
root=98:1 >uml_dir=/home/user-netkit/.netkit/mconsole
hosthome=/home/user-netkit quiet con0=fd:0,fd:1 con1=null SELINUX_INIT=0
>
and from cat /proc/cmdline
>
>name=pc1 title=pc1 mem=260M
ubd0=/home/user-netkit/Desktop/netkit/lab_netkit/eurecom_pmipv6_lab/pc1.disk,/home/user-netkit
>/Desktop/netkit/netkit/fs/netkit-fs root=98:1
uml_dir=/home/user-netkit/.netkit/mconsole
eth0=daemon,,,/home/user-netkit/.netkit
>/hubs/vhub_user-netkit_tap.cnct autoconf_eth0=192.168.1.112
def_route=192.168.1.111 eth1=daemon,,,/home/user-netkit/.netkit
>/hubs/vhub_user-netkit_WIFI1.cnct hosthome=/home/user-netkit
hostlab=/home/user-netkit/Desktop/netkit/lab_netkit
>/eurecom_pmipv6_lab
hostwd=/home/user-netkit/Desktop/netkit/lab_netkit/eurecom_pmipv6_lab quiet
con0=fd:0,fd:1 con1=null
>SELINUX_INIT=0
>
as you can see no "modules=" parameter is passed is this a fluke/glitch? and
can I possibly add it manually or where do I need to tweak in order to make
it automatic?
The /etc/rcS.d/S19mount-modules-dir file is there and I get what happens.
The $MODULES environment variable is empty because it didn't find the right
argument on the cmdline.
Best Regards,
Rui Costa
On Tue, Sep 20, 2011 at 4:44 PM, Massimo Rimondini <rimondin at dia.uniroma3.it
> wrote:
> **
> Dear Rui,
>
> >Modules directory not mounted.
>
>
> Strange. This means that virtual machines are not told to mount a kernel
> modules directory at all. This usually happens if no "modules=" parameter is
> passed on the UML kernel command line. Do you use vstart to launch your test
> virtual machines? Does the same vstart line plus the "--print" option show a
> "modules=" parameter? If the virtual machine manages to boot, could you
> check the output of "cat /proc/cmdline"?
>
> Here is a standard sample output of "vstart pc1 --print":
>
> ============= Starting virtual machine "pc1" =============
> Kernel: /home/max/netkit/kernel/netkit-kernel
> Modules: /home/max/netkit/kernel/modules
>
> You can see that vstart has recognized that a kernel modules directory is
> supposed to be used.
>
> Memory: 32 MB
> Model fs: /home/max/netkit/fs/netkit-fs
> Filesystem: /home/max/pc1.disk
> Hostfs at: /home/max
>
> Not running ==> xterm -e /home/max/netkit/kernel/netkit-kernel
> modules=/home/max/netkit/kernel/modules name=pc1 title=pc1 umid=pc1 mem=36M
> ubd0=/home/max/pc1.disk,/home/max/netkit/fs/netkit-fs root=98:1
> uml_dir=/home/max/.netkit/mconsole hosthome=/home/max quiet con0=fd:0,fd:1
> con1=null SELINUX_INIT=0
>
> ...and here the "modules=" statement specifies the directory on the host
> that contains the kernel modules.
>
> By the way inside the virtual machine the script
> /etc/rc3.d/S19mount-modules-dir
> doesn't exist.
>
>
> It's rcS.d, not rc3.d.
>
>
> Best Regards,
> Rui Costa
>
>
> Regards,
> Massimo
>
>
>
>
>
> On Mon, Sep 19, 2011 at 10:03 AM, Massimo Rimondini <
> rimondin at dia.uniroma3.it> wrote:
>
>> Dear Rui,
>>
>> then it must be the case that Netkit is attempting to mount the wrong path
>> under /lib/modules or is just failing to recognize that modules are required
>> inside the virtual machines.
>> You should see either one of these two messages during the virtual machine
>> boot phase:
>>
>> Mounting kernel modules directory /your/own/pa
>>
>> or:
>>
>> Modules directory not mounted.
>>
>> Which one do you see? Are they followed by an error?
>>
>> By the way, the task of mounting the modules directory is accomplished by
>> the /etc/rcS.d/S19mount-modules-dir script inside virtual machines.
>>
>> Regards,
>> Massimo
>>
>>
>> On 09/16/2011 05:24 PM, Costa >> wrote:
>>
>> Dear Massimo,
>>
>> Doing the command "mount | grep lib/modules" on the virtual machine
>> outputs nothing.
>> Makes sense since it did not load it.
>>
>> And yes, the path
>> "my/own/path/netkit/kernel/modules/lib/modules/2.6.28-netkit-K2.8/"
>> is there correctly and contains "modules.dep" file and several others...
>>
>> Regards,
>> Rui Costa
>>
>>
>> On Thu, Sep 15, 2011 at 3:32 PM, Massimo Rimondini <
>> rimondin at dia.uniroma3.it> wrote:
>>
>>> Dear Rui,
>>>
>>> As for the current issue I understand what you say.
>>> However the kernel modules and "modules.dep" files are in the correct
>>> location: /your/own/path/kernel/modules/lib/modules/2.6.28-netkit-K2.8/
>>>
>>>
>>> Fine.
>>>
>>> And despite this, the error output states:
>>> "Loading kernel modules...FATAL: Could not load
>>> /lib/modules/2.6.28-netkit-K2.8/modules.dep: No such file or directory"
>>> Which makes me think it actually is looking in /lib/modules/...
>>>
>>>
>>> Right. But that is the /lib/modules directory inside the virtual machine,
>>> which is mapped to a different path on the host. You can get the actual path
>>> by issuing the following command inside any virtual machine:
>>>
>>> pc1:~# mount | grep lib/modules
>>> none on /lib/modules type hostfs
>>> (rw,/home/max/netkit/kernel/modules/lib/modules)
>>>
>>>
>>> However, considering your reply, I see this is not really relevant, and
>>> other path issues should already have been ruled out by setting the
>>> CONFIG_LOCALVERSION parameter to "-netkit-K2.8".
>>>
>>> Could you check whether
>>> /your/own/path/kernel/modules/lib/modules/2.6.28-netkit-K2.8/ actually
>>> contains the modules.dep file?
>>> If not, you could be able to generate that file by hand by using the
>>> depmod tool.
>>>
>>> Regards,
>>> Massimo
>>>
>>>
>>>
>>>
>>> Best Regards,
>>> Rui Costa
>>>
>>>
>>>
>>>
>>> On Thu, Sep 15, 2011 at 2:02 PM, Massimo Rimondini <
>>> rimondin at dia.uniroma3.it> wrote:
>>>
>>>> Dear Rui,
>>>>
>>>> it would be interesting to know in more detail what you mean when you
>>>> say "still does not boot as a Netkit node".
>>>> However, if it's just a folder issue, it is likely easily solvable.
>>>> Consider that, when you tell Netkit to use a custom kernel in your/own/path,
>>>> compiled modules will be looked for in your/own/path/modules/lib/modules.
>>>> Does this help somehow?
>>>> The problem described in the post you mention should have been solved as
>>>> of Netkit release 2.8.
>>>>
>>>> Regards,
>>>> Massimo
>>>>
>>>> On 09/15/2011 12:07 PM, Costa >> wrote:
>>>>
>>>> Dear Massimo,
>>>>
>>>> Thank you very much for your help!
>>>> It seems that part of the problem is solved.
>>>> Although kernel.org is down I got the kernel sources from Linus
>>>> Torvalds github.
>>>> I tried a few kernels manually until I got one that booted as a netkit
>>>> node:
>>>>
>>>> => kernel v2.6.27.0 does the uml patch ok, has issues with an already
>>>> reported bug on vde_user.c, has a bug on making the kernel oldconfig
>>>> (already reported), but still does not boot as a netkit node.
>>>>
>>>> => kernel v2.6.31.0 has issues with the uml patch which be skipped,
>>>> has the same bug on making the kernel oldconfig (already reported), but
>>>> still does not boot as a netkit node.
>>>>
>>>> => kernel v2.6.28.0 has issues with the uml patch which be skipped, has
>>>> the same bug on making the kernel oldconfig (already reported), boots
>>>> _almost_ ok as netkit node.
>>>>
>>>> It has a fatal error on loading the "modules.dep" file from v2.6.28-0.
>>>> It seems that it can't find the file belonging to kernel v2.6.28.0-k2.8. It
>>>> seems that is searching on the wrong folder.. i'm not sure if it's because
>>>> i've been passing the kernel path as an argument on the vstart command (a
>>>> similar issue has been reported on
>>>> http://list.dia.uniroma3.it/pipermail/netkit.users/2010-August/000638.html
>>>> )
>>>> Any ideas on how to solve this modules.dep path ?
>>>>
>>>> Best regards,
>>>> Rui Costa
>>>>
>>>>
>>>>
>>>>
>>>> On Mon, Sep 12, 2011 at 2:57 PM, Massimo Rimondini <
>>>> rimondin at dia.uniroma3.it> wrote:
>>>>
>>>>> Dear Rui,
>>>>>
>>>>> not really useless. It may still be possible to apply them with minor
>>>>> offsets. I would give them a --dry-run try and see what happens.
>>>>> Or it may also be that the fixes/functionalities they provide are no
>>>>> longer necessary. Consider that about half of the provided patches are only
>>>>> required for MPLS support to work.
>>>>>
>>>>> Regards,
>>>>> Massimo
>>>>>
>>>>> On 09/12/2011 02:52 PM, Costa >> wrote:
>>>>>
>>>>> Dear Massimo,
>>>>>
>>>>> compiling for a newer kernel won't make the UML patch useless?
>>>>> I thought that the UML patch provided in netkit is specific for kernel
>>>>> 2.6.26.5.
>>>>> For a newer kernel the files are not the same or don't have the same
>>>>> content
>>>>> and therefore the same patch wouldn't work correct?
>>>>>
>>>>>
>>>>> Best Regards,
>>>>> Rui Costa
>>>>>
>>>>>
>>>>>
>>>>> On Sat, Sep 10, 2011 at 3:00 PM, Massimo Rimondini <
>>>>> rimondin at dia.uniroma3.it> wrote:
>>>>>
>>>>>> Dear Rui,
>>>>>>
>>>>>> could you try by recompiling a more recent version of the kernel?
>>>>>> The Makefile contains appropriate parameters to choose the release to
>>>>>> work on, see
>>>>>> https://github.com/maxonthegit/netkit-uml-kernel/blob/master/Makefile.devel#L35.
>>>>>> Most likely, you will need to disable MPLS support, which requires quite a
>>>>>> bit of tweaking of the Makefile itself (or just a round of kernel
>>>>>> compilation by hand).
>>>>>>
>>>>>> Regards,
>>>>>> Massimo
>>>>>>
>>>>>> Il 09/09/2011 16.25, Costa >> ha scritto:
>>>>>>
>>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> I'm new with netkit as well as with kernel building and I'm
>>>>>> having some trouble setting up a custom kernel for netkit.
>>>>>> I've tried both the ways that come in netkit's documentation
>>>>>> (automatic and manual) and tried a few more tweaks by myself.
>>>>>>
>>>>>> Compiling the customized kernel always goes OK, in whatever
>>>>>> way I do it, but the netkit's virtual machines crash on
>>>>>> booting. The trace results (you can check them at the end of
>>>>>> the e-mail below) are always extremely similar to another
>>>>>> issue reported on the netkit mailing list:
>>>>>>
>>>>>> http://list.dia.uniroma3.it/pipermail/netkit.users/2010-November/000686.html
>>>>>> I've also tried the approach explained in this thread but
>>>>>> ended up with the same result as before.
>>>>>>
>>>>>>
>>>>>> Does anyone have any hint on what is causing this or how to
>>>>>> solve it?
>>>>>>
>>>>>> Thanks in advance,
>>>>>> Best Regards,
>>>>>>
>>>>>> Rui Costa
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Here is the Netkit output:
>>>>>>
>>>>>> ============= Starting virtual machine "pc1" =============
>>>>>> Kernel:
>>>>>> /home/user-netkit/Desktop/netkit/custom-kernel/netkit/kernel/linu
>>>>>> x-2.6.26.5/linux
>>>>>> Memory: 32 MB
>>>>>> Model fs: /home/user-netkit/Desktop/netkit/netkit/fs/netkit-fs
>>>>>> Filesystem: /home/user-netkit/Desktop/netkit/pc1.disk (new)
>>>>>> Hostfs at: /home/user-netkit
>>>>>> Other args: sch_teql.max_equalizers=0
>>>>>>
>>>>>> Running ==>
>>>>>> /home/user-netkit/Desktop/netkit/custom-kernel/netkit/kernel/linux-2
>>>>>> .6.26.5/linux name=pc1 title=pc1 umid=pc1 mem=36M
>>>>>> ubd0=/home/user-netkit/Desktop
>>>>>> /netkit/pc1.disk,/home/user-netkit/Desktop/netkit/netkit/fs/netkit-fs
>>>>>> root=98:1
>>>>>> uml_dir=/home/user-netkit/.netkit/mconsole hosthome=/home/user-netkit
>>>>>> sch_teql.
>>>>>> max_equalizers=0 quiet con0=fd:0,fd:1 con1=null SELINUX_INIT=0
>>>>>> Locating the bottom of the address space ... 0x10000
>>>>>> Locating the top of the address space ... 0xc0000000
>>>>>> Core dump limits :
>>>>>> soft - 0
>>>>>> hard - NONE
>>>>>> 0xc0000000
>>>>>> Core dump limits :
>>>>>> soft - 0
>>>>>> hard - NONE
>>>>>> 0xc0000000
>>>>>> Core dump limits :
>>>>>> soft - 0
>>>>>> hard - NONE
>>>>>> 0xc0000000
>>>>>> Core dump limits :
>>>>>> soft - 0
>>>>>> hard - NONE
>>>>>> Checking for tmpfs mount on /dev/shm...OK
>>>>>> Checking PROT_EXEC mmap in /dev/shm/...OK
>>>>>> OK
>>>>>> OK
>>>>>> OK
>>>>>> UML running in SKAS0 mode
>>>>>> Adding 2260992 bytes to physical memory to account for exec-shield gap
>>>>>> Checking that ptrace can change system call numbers...OK
>>>>>> Checking syscall emulation patch for ptrace...OK
>>>>>> Checking advanced syscall emulation patch for ptrace...OK
>>>>>> Checking for the skas3 patch in the host:
>>>>>> - /proc/mm...not found: No such file or directory
>>>>>> - PTRACE_FAULTINFO...not found
>>>>>> - PTRACE_LDT...not found
>>>>>> *** buffer overflow detected ***:
>>>>>> /home/user-netkit/Desktop/netkit/custom-kernel
>>>>>> /netkit/kernel/linux-2.6.26.5/linux terminated
>>>>>> ======= Backtrace: =========
>>>>>> /lib/tls/i686/cmov/libc.so.6(__fortify_fail+0x50)[0x1f2390]
>>>>>> /lib/tls/i686/cmov/libc.so.6(+0xe12ca)[0x1f12ca]
>>>>>> /lib/tls/i686/cmov/libc.so.6(+0xe0c29)[0x1f0c29]
>>>>>> /lib/tls/i686/cmov/libc.so.6(__snprintf_chk+0x34)[0x1f0b14]
>>>>>>
>>>>>> /home/user-netkit/Desktop/netkit/custom-kernel/netkit/kernel/linux-2.6.26.5/linu
>>>>>> x[0x808cdee]
>>>>>> [0x0]
>>>>>> ======= Memory map: ========
>>>>>> 00110000-00263000 r-xp 00000000 68:01 9438224
>>>>>> /lib/tls/i686/cmov/libc-2.11.1.
>>>>>> so
>>>>>> 00263000-00264000 ---p 00153000 68:01 9438224
>>>>>> /lib/tls/i686/cmov/libc-2.11.1.
>>>>>> so
>>>>>> 00264000-00266000 r-xp 00153000 68:01 9438224
>>>>>> /lib/tls/i686/cmov/libc-2.11.1.
>>>>>> so
>>>>>> 00266000-00267000 rwxp 00155000 68:01 9438224
>>>>>> /lib/tls/i686/cmov/libc-2.11.1.
>>>>>> so
>>>>>> 00267000-0026a000 rwxp 00000000 00:00 0
>>>>>> 0026a000-00272000 r-xp 00000000 68:01 9438245
>>>>>> /lib/tls/i686/cmov/libnss_nis-2
>>>>>> .11.1.so
>>>>>> 00272000-00273000 r-xp 00007000 68:01 9438245
>>>>>> /lib/tls/i686/cmov/libnss_nis-2
>>>>>> .11.1.so
>>>>>> 00273000-00274000 rwxp 00008000 68:01 9438245
>>>>>> /lib/tls/i686/cmov/libnss_nis-2
>>>>>> .11.1.so
>>>>>> 00276000-00291000 r-xp 00000000 68:01 9306137 /lib/ld-2.11.1.so
>>>>>> 00291000-00292000 r-xp 0001a000 68:01 9306137 /lib/ld-2.11.1.so
>>>>>> 00292000-00293000 rwxp 0001b000 68:01 9306137 /lib/ld-2.11.1.so
>>>>>> 00402000-00404000 rwxp 00000000 00:00 0
>>>>>> 00429000-00446000 r-xp 00000000 68:01 9306195 /lib/libgcc_s.so.1
>>>>>> 00446000-00447000 r-xp 0001c000 68:01 9306195 /lib/libgcc_s.so.1
>>>>>> 00447000-00448000 rwxp 0001d000 68:01 9306195 /lib/libgcc_s.so.1
>>>>>> 004f9000-004fc000 rwxp 00000000 00:00 0
>>>>>> 00681000-00694000 r-xp 00000000 68:01 9438235
>>>>>> /lib/tls/i686/cmov/libnsl-2.11.
>>>>>> 1.so
>>>>>> 00694000-00695000 r-xp 00012000 68:01 9438235
>>>>>> /lib/tls/i686/cmov/libnsl-2.11.
>>>>>> 1.so
>>>>>> 00695000-00696000 rwxp 00013000 68:01 9438235
>>>>>> /lib/tls/i686/cmov/libnsl-2.11.
>>>>>> 1.so
>>>>>> 00696000-00698000 rwxp 00000000 00:00 0
>>>>>> 0072f000-00730000 rwxp 00000000 00:00 0
>>>>>> 00747000-00748000 rwxp 00000000 00:00 0
>>>>>> 008a2000-008a3000 r-xp 00000000 00:00 0 [vdso]
>>>>>> 009c0000-009ca000 r-xp 00000000 68:01 9438241
>>>>>> /lib/tls/i686/cmov/libnss_files
>>>>>> -2.11.1.so
>>>>>> 009ca000-009cb000 r-xp 00009000 68:01 9438241
>>>>>> /lib/tls/i686/cmov/libnss_files
>>>>>> -2.11.1.so
>>>>>> 009cb000-009cc000 rwxp 0000a000 68:01 9438241
>>>>>> /lib/tls/i686/cmov/libnss_files
>>>>>> -2.11.1.so
>>>>>> 00d36000-00d37000 rwxp 00000000 00:00 0
>>>>>> 00d9b000-00d9d000 r-xp 00000000 68:01 9438258
>>>>>> /lib/tls/i686/cmov/libutil-2.11
>>>>>> .1.so
>>>>>> 00d9d000-00d9e000 r-xp 00001000 68:01 9438258
>>>>>> /lib/tls/i686/cmov/libutil-2.11
>>>>>> .1.so
>>>>>> 00d9e000-00d9f000 rwxp 00002000 68:01 9438258
>>>>>> /lib/tls/i686/cmov/libutil-2.11
>>>>>> .1.so
>>>>>> 00e05000-00e0b000 r-xp 00000000 68:01 9438237
>>>>>> /lib/tls/i686/cmov/libnss_compa
>>>>>> t-2.11.1.so
>>>>>> 00e0b000-00e0c000 r-xp 00006000 68:01 9438237
>>>>>> /lib/tls/i686/cmov/libnss_compa
>>>>>> t-2.11.1.so
>>>>>> 00e0c000-00e0d000 rwxp 00007000 68:01 9438237
>>>>>> /lib/tls/i686/cmov/libnss_compa
>>>>>> t-2.11.1.so
>>>>>> 08048000-08422000 rwxp 00000000 68:01 12327718
>>>>>> /home/user-netkit/Desktop/netki
>>>>>> t/custom-kernel/netkit/kernel/linux-2.6.26.5/linux
>>>>>> 08422000-08446000 rwxp 00000000 00:00 0
>>>>>> 0864d000-0866e000 rwxp 00000000 00:00 0 [heap]
>>>>>> 0866e000-0a670000 rwxs 00626000 00:10 8225
>>>>>> /dev/shm/vm_file-RhLyFA (delete
>>>>>> d)
>>>>>> bf8fb000-bf910000 rwxp 00000000 00:00 0 [stack]
>>>>>> Terminated
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Netkit.users mailing listNetkit.users at list.dia.uniroma3.ithttp://list.dia.uniroma3.it/mailman/listinfo/netkit.users
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Netkit.users mailing list
>>>>>> Netkit.users at list.dia.uniroma3.it
>>>>>> http://list.dia.uniroma3.it/mailman/listinfo/netkit.users
>>>>>>
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Netkit.users mailing listNetkit.users at list.dia.uniroma3.ithttp://list.dia.uniroma3.it/mailman/listinfo/netkit.users
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Netkit.users mailing list
>>>>> Netkit.users at list.dia.uniroma3.it
>>>>> http://list.dia.uniroma3.it/mailman/listinfo/netkit.users
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> Netkit.users mailing listNetkit.users at list.dia.uniroma3.ithttp://list.dia.uniroma3.it/mailman/listinfo/netkit.users
>>>>
>>>>
>>>> _______________________________________________
>>>> Netkit.users mailing list
>>>> Netkit.users at list.dia.uniroma3.it
>>>> http://list.dia.uniroma3.it/mailman/listinfo/netkit.users
>>>>
>>>>
>>>
>>> _______________________________________________
>>> Netkit.users mailing listNetkit.users at list.dia.uniroma3.ithttp://list.dia.uniroma3.it/mailman/listinfo/netkit.users
>>>
>>>
>>> _______________________________________________
>>> Netkit.users mailing list
>>> Netkit.users at list.dia.uniroma3.it
>>> http://list.dia.uniroma3.it/mailman/listinfo/netkit.users
>>>
>>>
>>
>> _______________________________________________
>> Netkit.users mailing listNetkit.users at list.dia.uniroma3.ithttp://list.dia.uniroma3.it/mailman/listinfo/netkit.users
>>
>>
>> _______________________________________________
>> Netkit.users mailing list
>> Netkit.users at list.dia.uniroma3.it
>> http://list.dia.uniroma3.it/mailman/listinfo/netkit.users
>>
>>
>
> _______________________________________________
> Netkit.users mailing listNetkit.users at list.dia.uniroma3.ithttp://list.dia.uniroma3.it/mailman/listinfo/netkit.users
>
>
> _______________________________________________
> Netkit.users mailing list
> Netkit.users at list.dia.uniroma3.it
> http://list.dia.uniroma3.it/mailman/listinfo/netkit.users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://list.dia.uniroma3.it/pipermail/netkit.users/attachments/20110920/31164c12/attachment-0001.html>
More information about the Netkit.users
mailing list