mirror of
https://git.rtems.org/rtems-libbsd/
synced 2025-05-13 11:39:18 +08:00
549 lines
18 KiB
Plaintext
549 lines
18 KiB
Plaintext
RTEMS-NFS
|
|
=========
|
|
|
|
A NFS-V2 client implementation for the RTEMS real-time
|
|
executive.
|
|
|
|
Author: Till Straumann <strauman@slac.stanford.edu>, 2002
|
|
|
|
Copyright 2002, Stanford University and
|
|
Till Straumann <strauman@slac.stanford.edu>
|
|
|
|
Stanford Notice
|
|
***************
|
|
|
|
Acknowledgement of sponsorship
|
|
* * * * * * * * * * * * * * * *
|
|
This software was produced by the Stanford Linear Accelerator Center,
|
|
Stanford University, under Contract DE-AC03-76SFO0515 with the Department
|
|
of Energy.
|
|
|
|
|
|
Contents
|
|
--------
|
|
I Overview
|
|
1) Performance
|
|
2) Reference Platform / Test Environment
|
|
|
|
II Usage
|
|
1) Initialization
|
|
2) Mounting Remote Server Filesystems
|
|
3) Unmounting
|
|
4) Unloading
|
|
5) Dumping Information / Statistics
|
|
|
|
III Implementation Details
|
|
1) RPCIOD
|
|
2) NFS
|
|
3) RTEMS Resources Used By NFS/RPCIOD
|
|
4) Caveats & Bugs
|
|
|
|
IV Licensing & Disclaimers
|
|
|
|
I Overview
|
|
-----------
|
|
|
|
This package implements a simple non-caching NFS
|
|
client for RTEMS. Most of the system calls are
|
|
supported with the exception of 'mount', i.e. it
|
|
is not possible to mount another FS on top of NFS
|
|
(mostly because of the difficulty that arises when
|
|
mount points are deleted on the server). It
|
|
shouldn't be hard to do, though.
|
|
|
|
Note: this client supports NFS vers. 2 / MOUNT vers. 1;
|
|
NFS Version 3 or higher are NOT supported.
|
|
|
|
The package consists of two modules: RPCIOD and NFS
|
|
itself.
|
|
|
|
- RPCIOD is a UDP/RPC multiplexor daemon. It takes
|
|
RPC requests from multiple local client threads,
|
|
funnels them through a single socket to multiple
|
|
servers and dispatches the replies back to the
|
|
(blocked) requestor threads.
|
|
RPCIOD does packet retransmission and handles
|
|
timeouts etc.
|
|
Note however, that it does NOT do any XDR
|
|
marshalling - it is up to the requestor threads
|
|
to do the XDR encoding/decoding. RPCIOD _is_ RPC
|
|
specific, though, because its message dispatching
|
|
is based on the RPC transaction ID.
|
|
|
|
- The NFS package maps RTEMS filesystem calls
|
|
to proper RPCs, it does the XDR work and
|
|
hands marshalled RPC requests to RPCIOD.
|
|
All of the calls are synchronous, i.e. they
|
|
block until they get a reply.
|
|
|
|
1) Performance
|
|
- - - - - - - -
|
|
Performance sucks (due to the lack of
|
|
readahead/delayed write and caching). On a fast
|
|
(100Mb/s) ethernet, it takes about 20s to copy a
|
|
10MB file from NFS to NFS. I found, however, that
|
|
vxWorks' NFS client doesn't seem to be any
|
|
faster...
|
|
|
|
Since there is no buffer cache with read-ahead
|
|
implemented, all NFS reads are synchronous RPC
|
|
calls. Every read operation involves sending a
|
|
request and waiting for the reply. As long as the
|
|
overhead (sending request + processing it on the
|
|
server) is significant compared to the time it
|
|
takes to transferring the actual data, increasing
|
|
the amount of data per request results in better
|
|
throughput. The UDP packet size limit imposes a
|
|
limit of 8k per RPC call, hence reading from NFS
|
|
in chunks of 8k is better than chunks of 1k [but
|
|
chunks >8k are not possible, i.e., simply not
|
|
honoured: read(a_nfs_fd, buf, 20000) returns
|
|
8192]. This is similar to the old linux days
|
|
(mount with rsize=8k). You can let stdio take
|
|
care of the buffering or use 8k buffers with
|
|
explicit read(2) operations. Note that stdio
|
|
honours the file-system's st_blksize field
|
|
if newlib is compiled with HAVE_BLKSIZE defined.
|
|
In this case, stdio uses 8k buffers for files
|
|
on NFS transparently. The blocksize NFS
|
|
reports can be tuned with a global variable
|
|
setting (see nfs.c for details).
|
|
|
|
Further increase of throughput can be achieved
|
|
with read-ahead (issuing RPC calls in parallel
|
|
[send out request for block n+1 while you are
|
|
waiting for data of block n to arrive]). Since
|
|
this is not handled by the file system itself, you
|
|
would have to code this yourself e.g., using
|
|
parallel threads to read from a single file from
|
|
interleaved offsets.
|
|
|
|
Another obvious improvement can be achieved if
|
|
processing the data takes a significant amount of
|
|
time. Then, having a pipeline of threads for
|
|
reading data and processing them makes sense
|
|
[thread b processes chunk n while thread a blocks
|
|
in read(chunk n+1)].
|
|
|
|
Some performance figures:
|
|
Software: src/nfsTest.c:nfsReadTest() [data not
|
|
processed in any way].
|
|
Hardware: MVME6100
|
|
Network: 100baseT-FD
|
|
Server: Linux-2.6/RHEL4-smp [dell precision 420]
|
|
File: 10MB
|
|
|
|
Results:
|
|
Single threaded ('normal') NFS read, 1k buffers: 3.46s (2.89MB/s)
|
|
Single threaded ('normal') NFS read, 8k buffers: 1.31s (7.63MB/s)
|
|
Multi threaded; 2 readers, 8k buffers/xfers: 1.12s (8.9 MB/s)
|
|
Multi threaded; 3 readers, 8k buffers/xfers: 1.04s (9.6 MB/s)
|
|
|
|
2) Reference Platform
|
|
- - - - - - - - - - -
|
|
RTEMS-NFS was developed and tested on
|
|
|
|
o RTEMS-ss20020301 (local patches applied)
|
|
o PowerPC G3, G4 on Synergy SVGM series board
|
|
(custom 'SVGM' BSP, to be released soon)
|
|
o PowerPC 604 on MVME23xx
|
|
(powerpc/shared/motorola-powerpc BSP)
|
|
o Test Environment:
|
|
- RTEMS executable running CEXP
|
|
- rpciod/nfs dynamically loaded from TFTPfs
|
|
- EPICS application dynamically loaded from NFS;
|
|
the executing IOC accesses all of its files
|
|
on NFS.
|
|
|
|
II Usage
|
|
---------
|
|
|
|
After linking into the system and proper initialization
|
|
(rtems-NFS supports 'magic' module initialization when
|
|
loaded into a running system with the CEXP loader),
|
|
you are ready for mounting NFSes from a server
|
|
(I avoid the term NFS filesystem because NFS already
|
|
stands for 'Network File System').
|
|
|
|
You should also read the
|
|
|
|
- "RTEMS Resources Used By NFS/RPCIOD"
|
|
- "CAVEATS & BUGS"
|
|
|
|
below.
|
|
|
|
1) Initialization
|
|
- - - - - - - - -
|
|
NFS consists of two modules who must be initialized:
|
|
|
|
a) the RPCIO daemon package; by calling
|
|
|
|
rpcUdpInit();
|
|
|
|
note that this step must be performed prior to
|
|
initializing NFS:
|
|
|
|
b) NFS is initialized by calling
|
|
|
|
nfsInit( smallPoolDepth, bigPoolDepth );
|
|
|
|
if you supply 0 (zero) values for the pool
|
|
depths, the compile-time default configuration
|
|
is used which should work fine.
|
|
|
|
NOTE: when using CEXP to load these modules into a
|
|
running system, initialization will be performed
|
|
automagically.
|
|
|
|
2) Mounting Remote Server Filesystems
|
|
- - - - - - - - - - - - - - - - - - -
|
|
|
|
There are two interfaces for mounting an NFS:
|
|
|
|
- The (non-POSIX) RTEMS 'mount()' call:
|
|
|
|
mount( &mount_table_entry_pointer,
|
|
&filesystem_operations_table_pointer,
|
|
options,
|
|
device,
|
|
mount_point )
|
|
|
|
Note that you must specify a 'mount_table_entry_pointer'
|
|
(use a dummy) - RTEMS' mount() doesn't grok a NULL for
|
|
the first argument.
|
|
|
|
o for the 'filesystem_operations_table_pointer', supply
|
|
|
|
&nfs_fs_ops
|
|
|
|
o options are constants (see RTEMS headers) for specifying
|
|
read-only / read-write mounts.
|
|
|
|
o the 'device' string specifies the remote filesystem
|
|
who is to be mounted. NFS expects a string conforming
|
|
to the following format (EBNF syntax):
|
|
|
|
[ <uid> '.' <gid> '@' ] <hostip> ':' <path>
|
|
|
|
The first optional part of the string allows you
|
|
to specify the credentials to be used for all
|
|
subsequent transactions with this server. If the
|
|
string is omitted, the EUID/EGID of the executing
|
|
thread (i.e. the thread performing the 'mount' -
|
|
NFS will still 'remember' these values and use them
|
|
for all future communication with this server).
|
|
|
|
The <hostip> part denotes the server IP address
|
|
in standard 'dot' notation. It is followed by
|
|
a colon and the (absolute) path on the server.
|
|
Note that no extra characters or whitespace must
|
|
be present in the string. Example 'device' strings
|
|
are:
|
|
|
|
"300.99@192.168.44.3:/remote/rtems/root"
|
|
|
|
"192.168.44.3:/remote/rtems/root"
|
|
|
|
o the 'mount_point' string identifies the local
|
|
directory (most probably on IMFS) where the NFS
|
|
is to be mounted. Note that the mount point must
|
|
already exist with proper permissions.
|
|
|
|
- Alternate 'mount' interface. NFS offers a more
|
|
convenient wrapper taking three string arguments:
|
|
|
|
nfsMount(uidgid_at_host, server_path, mount_point)
|
|
|
|
This interface does DNS lookup (see reentrancy note
|
|
below) and creates the mount point if necessary.
|
|
|
|
o the first argument specifies the server and
|
|
optionally the uid/gid to be used for authentication.
|
|
The semantics are exactly as described above:
|
|
|
|
[ <uid> '.' <gid> '@' ] <host>
|
|
|
|
The <host> part may be either a host _name_ or
|
|
an IP address in 'dot' notation. In the former
|
|
case, nfsMount() uses 'gethostbyname()' to do
|
|
a DNS lookup.
|
|
|
|
IMPORTANT NOTE: gethostbyname() is NOT reentrant/
|
|
thread-safe and 'nfsMount()' (if not provided with an
|
|
IP/dot address string) is hence subject to race conditions.
|
|
|
|
o the 'server_path' and 'mount_point' arguments
|
|
are described above.
|
|
NOTE: If the mount point does not exist yet,
|
|
nfsMount() tries to create it.
|
|
|
|
o if nfsMount() is called with a NULL 'uidgid_at_host'
|
|
argument, it lists all currently mounted NFS
|
|
|
|
3) Unmounting
|
|
- - - - - - -
|
|
An NFS can be unmounted using RTEMS 'unmount()'
|
|
call (yep, it is unmount() - not umount()):
|
|
|
|
unmount(mount_point)
|
|
|
|
Note that you _must_ supply the mount point (string
|
|
argument). It is _not_ possible to specify the
|
|
'mountee' when unmounting. NFS implements no
|
|
convenience wrapper for this (yet), essentially because
|
|
(although this sounds unbelievable) it is non-trivial
|
|
to lookup the path leading to an RTEMS filesystem
|
|
directory node.
|
|
|
|
4) Unloading
|
|
- - - - - - -
|
|
After unmounting all NFS from the system, the NFS
|
|
and RPCIOD modules may be stopped and unloaded.
|
|
Just call 'nfsCleanup()' and 'rpcUdpCleanup()'
|
|
in this order. You should evaluate the return value
|
|
of these routines which is non-zero if either
|
|
of them refuses to yield (e.g. because there are
|
|
still mounted filesystems).
|
|
Again, when unloading is done by CEXP this is
|
|
transparently handled.
|
|
|
|
5) Dumping Information / Statistics
|
|
- - - - - - - - - - - - - - - - - -
|
|
|
|
Rudimentary RPCIOD statistics are printed
|
|
to a file (stdout when NULL) by
|
|
|
|
int rpcUdpStats(FILE *f)
|
|
|
|
A list of all currently mounted NFS can be
|
|
printed to a file (stdout if NULL) using
|
|
|
|
int nfsMountsShow(FILE *f)
|
|
|
|
For convenience, this routine is also called
|
|
by nfsMount() when supplying NULL arguments.
|
|
|
|
III Implementation Details
|
|
--------------------------
|
|
|
|
1) RPCIOD
|
|
- - - - -
|
|
|
|
RPCIOD was created to
|
|
|
|
a) avoid non-reentrant librpc calls.
|
|
b) support 'asynchronous' operation over a single
|
|
socket.
|
|
|
|
RPCIOD is a daemon thread handling 'transaction objects'
|
|
(XACTs) through an UDP socket. XACTs are marshalled RPC
|
|
calls/replies associated with RPC servers and requestor
|
|
threads.
|
|
|
|
requestor thread: network:
|
|
|
|
XACT packet
|
|
| |
|
|
V V
|
|
| message queue | ( socket )
|
|
| | ^
|
|
----------> <----- | |
|
|
RPCIOD |
|
|
/ --------------
|
|
timeout/ (re) transmission
|
|
|
|
|
|
A requestor thread drops a transaction into
|
|
the message queue and goes to sleep. The XACT is
|
|
picked up by rpciod who is listening for events from
|
|
three sources:
|
|
|
|
o the request queue
|
|
o packet arrival at the socket
|
|
o timeouts
|
|
|
|
RPCIOD sends the XACT to its destination server and
|
|
enqueues the pending XACT into an ordered list of
|
|
outstanding transactions.
|
|
|
|
When a packet arrives, RPCIOD (based on the RPC transaction
|
|
ID) looks up the matching XACT and wakes up the requestor
|
|
who can then XDR-decode the RPC results found in the XACT
|
|
object's buffer.
|
|
|
|
When a timeout expires, RPCIOD examines the outstanding
|
|
XACT that is responsible for the timeout. If its lifetime
|
|
has not expired yet, RPCIOD resends the request. Otherwise,
|
|
the XACT's error status is set and the requestor is woken up.
|
|
|
|
RPCIOD dynamically adjusts the retransmission intervals
|
|
based on the average round-trip time measured (on a per-server
|
|
basis).
|
|
|
|
Having the requestors event driven (rather than blocking
|
|
e.g. on a semaphore) is geared to having many different
|
|
requestors (one synchronization object per requestor would
|
|
be needed otherwise).
|
|
|
|
Requestors who want to do asynchronous IO need a different
|
|
interface which will be added in the future.
|
|
|
|
1.a) Reentrancy
|
|
- - - - - - - -
|
|
RPCIOD does no non-reentrant librpc calls.
|
|
|
|
1.b) Efficiency
|
|
- - - - - - - -
|
|
We shouldn't bother about efficiency until pipelining (read-ahead/
|
|
delayed write) and caching are implemented. The round-trip delay
|
|
associated with every single RPC transaction clearly is a big
|
|
performance killer.
|
|
|
|
Nevertheless, I could not withstand the temptation to eliminate
|
|
the extra copy step involved with socket IO:
|
|
|
|
A user data object has to be XDR encoded into a buffer. The
|
|
buffer given to the socket where it is copied into MBUFs.
|
|
(The network chip driver might even do more copying).
|
|
|
|
Likewise, on reception 'recvfrom' copies MBUFS into a user
|
|
buffer which is XDR decoded into the final user data object.
|
|
|
|
Eliminating the copying into (possibly multiple) MBUFS by
|
|
'sendto()' is actually a piece of cake. RPCIOD uses the
|
|
'sosend()' routine [properly wrapped] supplying a single
|
|
MBUF header who directly points to the marshalled buffer
|
|
:-)
|
|
|
|
Getting rid of the extra copy on reception was (only a little)
|
|
harder: I derived a 'XDR-mbuf' stream from SUN's xdr_mem which
|
|
allows for XDR-decoding out of a MBUF chain who is obtained by
|
|
soreceive().
|
|
|
|
2) NFS
|
|
- - - -
|
|
The actual NFS implementation is straightforward and essentially
|
|
'passive' (no threads created). Any RTEMS task executing a
|
|
filesystem call dispatched to NFS (such as 'opendir()', 'lseek()'
|
|
or 'unlink()') ends up XDR encoding arguments, dropping a
|
|
XACT into RPCIOD's message queue and going to sleep.
|
|
When woken up by RPCIOD, the XACT is decoded (using the XDR-mbuf
|
|
stream mentioned above) and the properly cooked-up results are
|
|
returned.
|
|
|
|
3) RTEMS Resources Used By NFS/RPCIOD
|
|
- - - - - - - - - - - - - - - - - - -
|
|
|
|
The RPCIOD/NFS package uses the following resources. Some
|
|
parameters are compile-time configurable - consult the
|
|
source files for details.
|
|
|
|
RPCIOD:
|
|
o 1 task
|
|
o 1 message queue
|
|
o 1 socket/filedescriptor
|
|
o 2 semaphores (a third one is temporarily created during
|
|
rpcUdpCleanup()).
|
|
o 1 RTEMS EVENT (by default RTEMS_EVENT_30).
|
|
IMPORTANT: this event is used by _every_ thread executing
|
|
NFS system calls and hence is RESERVED.
|
|
o 3 events only used by RPCIOD itself, i.e. these must not
|
|
be sent to RPCIOD by no other thread (except for the intended
|
|
use, of course). The events involved are 1,2,3.
|
|
o preemption disabled sections: NONE
|
|
o sections with interrupts disabled: NONE
|
|
o NO 'timers' are used (timer code would run in IRQ context)
|
|
o memory usage: n.a
|
|
|
|
NFS:
|
|
o 2 message queues
|
|
o 2 semaphores
|
|
o 1 semaphore per mounted NFS
|
|
o 1 slot in driver entry table (for major number)
|
|
o preemption disabled sections: NONE
|
|
o sections with interrupts disabled: NONE
|
|
o 1 task + 1 semaphore temporarily created when
|
|
listing mounted filesystems (rtems_filesystem_resolve_location())
|
|
|
|
4) CAVEATS & BUGS
|
|
- - - - - - - - -
|
|
Unfortunately, some bugs crawl around in the filesystem generics.
|
|
(Some of them might already be fixed in versions later than
|
|
rtems-ss-20020301).
|
|
I recommend to use the patch distributed with RTEMS-NFS.
|
|
|
|
o RTEMS uses/used (Joel said it has been fixed already) a 'short'
|
|
ino_t which is not enough for NFS.
|
|
The driver detects this problem and enables a workaround. In rare
|
|
situations (mainly involving 'getcwd()' improper inode comparison
|
|
may result (due to the restricted size, stat() returns st_ino modulo
|
|
2^16). In most cases, however, st_dev is compared along with st_ino
|
|
which will give correct results (different files may yield identical
|
|
st_ino but they will have different st_dev). However, there is
|
|
code (in getcwd(), for example) who assumes that files residing
|
|
in one directory must be hosted by the same device and hence omits
|
|
the st_dev comparison. In such a case, the workaround will fail.
|
|
|
|
NOTE: changing the size (sys/types.h) of ino_t from 'short' to 'long'
|
|
is strongly recommended. It is NOT included in the patch, however
|
|
as this is a major change requiring ALL of your sources to
|
|
be recompiled.
|
|
|
|
THE ino_t SIZE IS FIXED IN GCC-3.2/NEWLIB-1.10.0-2 DISTRIBUTED BY
|
|
OAR.
|
|
|
|
o You may work around most filesystem bugs by observing the following
|
|
rules:
|
|
|
|
* never use chroot() (fixed by the patch)
|
|
* never use getpwent(), getgrent() & friends - they are NOT THREAD
|
|
safe (fixed by the patch)
|
|
* NEVER use rtems_libio_share_private_env() - not even with the
|
|
patch applied. Just DONT - it is broken by design.
|
|
* All threads who have their own userenv (who have called
|
|
rtems_libio_set_private_env()) SHOULD 'chdir("/")' before
|
|
terminating. Otherwise, (i.e. if their cwd is on NFS), it will
|
|
be impossible to unmount the NFS involved.
|
|
|
|
o The patch slightly changes the semantics of 'getpwent()' and
|
|
'getgrent()' & friends (to what is IMHO correct anyways - the patch is
|
|
also needed to fix another problem, however): with the patch applied,
|
|
the passwd and group files are always accessed from the 'current' user
|
|
environment, i.e. a thread who has changed its 'root' or 'uid' might
|
|
not be able to access these files anymore.
|
|
|
|
o NOTE: RTEMS 'mount()' / 'unmount()' are NOT THREAD SAFE.
|
|
|
|
o The NFS protocol has no 'append' or 'seek_end' primitive. The client
|
|
must query the current file size (this client uses cached info) and
|
|
change the local file pointer accordingly (in 'O_APPEND' mode).
|
|
Obviously, this involves a race condition and hence multiple clients
|
|
writing the same file may lead to corruption.
|
|
|
|
IV Licensing & Disclaimers
|
|
--------------------------
|
|
|
|
NFS is distributed under the SLAC License - consult the
|
|
separate 'LICENSE' file.
|
|
|
|
Government disclaimer of liability
|
|
- - - - - - - - - - - - - - - - -
|
|
Neither the United States nor the United States Department of Energy,
|
|
nor any of their employees, makes any warranty, express or implied,
|
|
or assumes any legal liability or responsibility for the accuracy,
|
|
completeness, or usefulness of any data, apparatus, product, or process
|
|
disclosed, or represents that its use would not infringe privately
|
|
owned rights.
|
|
|
|
Stanford disclaimer of liability
|
|
- - - - - - - - - - - - - - - - -
|
|
Stanford University makes no representations or warranties, express or
|
|
implied, nor assumes any liability for the use of this software.
|
|
|
|
Maintenance of notice
|
|
- - - - - - - - - - -
|
|
In the interest of clarity regarding the origin and status of this
|
|
software, Stanford University requests that any recipient of it maintain
|
|
this notice affixed to any distribution by the recipient that contains a
|
|
copy or derivative of this software.
|