FX::QMemMap Class Reference
[File type i/o devices]

#include <QMemMap.h>

Inheritance diagram for FX::QMemMap:

List of all members.

---

Detailed Description

An i/o device accessing mapped memory.

Mapped memory is conceptually a portion of a disc-based file indirectly accessed as though it were a region of memory. All modern operating systems support this facility, yet astonishingly little use is made of it in most computer programs. Quite possibly this is because it is harder to use than it should be which is a shame because it makes available the very best file access caching and buffering to your program completely transparently.

QMemMap addresses this by providing transparent usage of mapped memory and portably too. There are no issues with accidentally running off the end of the mapped section nor off the start. Use as shared memory between processes is also ridiculously easy - you simply name the device the same in both processes and voilá, you're now working with the same patch of memory! Remember to synchronise multiple process access - the most portable is via msgs using a pipe, however if there is low contention in access it can be done via FX::QShrdMemMutex.

You can map in an existing FX::QFile device and indeed internally QMemMap creates one if you don't specify a FX::QFile but do a filename. You should note that as a result, QMemMap can lose track of the real file length under the same conditions as FX::QFile for which reloadSize() is also provided. If you choose shared memory (QMemMap::Memory) rather than a filename (QMemMap::File) then the file used is kept in memory as much as possible rather than being flushed to disc as soon as possible.

If you wish to convert a FX::QBuffer into a QMemMap, simply create an FX::FXStream refering to the QMemMap and do

FXStream ds((QMemMap) x);
ds << (QBuffer) y;

QMemMap is heavily optimised including custom getch() and putch() especially for mostly sequential transfers (where at() is not moving more than FXProcess::pageSize() at a time). Also file changes moving forwards are better than backwards. In ideal conditions, not much more than a memcpy() is being performed so the bigger the chunks you give it, the better.

Default security is for FX::FXACLEntity::everything() to have full access when the type is mapped memory - file maps deny everything access except the local process. This makes sense as mapped memory is generally for inter-process communication - if however it's a private region for communication with a known process it makes sense to FXRESTRICT access to the user running that process at least. Note that until the map is opened, permissions() returns what will be applied to the map on open() rather than the map itself - if you want the latter, use the static method. Note also that the map's security is not linked in any way with that of any file underlying it - you must set them separately. Note that on POSIX, the underlying file's permissions are those for the map and setting them here has no effect - however, if it's shared memory then the permissions do have an effect.

Like FX::QPipe, the shared memory name is deleted by its creator on POSIX only (on Windows it lasts until the last thing referring to it closes). If you don't want this, specify IO_DontUnlink in the flags to open().

Usage:

Simply instantiate, set filename or name and open(). Once open, use mapIn() to map in a specified section and mapOut() to map it out. On 32 bit architectures, files bigger than ~1Gb on some operating systems will fail to map in so be aware of the consequences of the defaults in mapIn() which map everything in. Failures to map in due to address space exhaustion cause mapIn() to return zero rather than an exception - however, the mapping remains on the books and may get mapped in silently the next time something is mapped out. See FX::FXProcess::virtualAddrSpaceLeft().

This is done this way because the other important facility QMemMap offers is transparently using the mapped section(s) when the file pointer is within one and doing normal readBlock() and writeBlock() with the FX::QFile when outside. Thus you can afford to map in the most commonly used portion(s) of your file and let the rest happen via buffered i/o as usual.

Warning:

You really do not want to read or write parts of a file which have been mapped in using FX::QFile directly. This inevitably causes data corruption. To avoid this, open() calls flush() on its FX::QFile if it's already open so therefore after this point, all i/o on the file should be done via QMemMap.

Indeed unless you need absolutely critical sustained sequential reads or writes, I'd recommend doing all your file i/o via FX::QMemMap instead of via FX::QFile and indeed I've made the non static methods exactly the same. Memory mapped files are demand-read often with intelligent prefetching as read accesses are performed from its mapped section. Writes almost always remain in the RAM backing the mapped section until the system is idle (ie; not doing more reads) so you get the maximum performance from your hardware. Even better when free memory is tight, the operating system will write out dirty pages sooner rather than later thus getting the best of all possible worlds. On my development system running Win2k I get a three-fold speed increase copying one file to another over QFile and that's even with NTFS's excellent caching of buffered i/o.

Note:

I've found that memory mapped file i/o is considerably faster on Linux when using ReiserFS rather than ext3.

Extending the file size with truncate() works as expected - however depending on the host OS, it can require recreating the map which implies deletion and recreation of all mapped sections (which may move position). Shortening the file requires wholly unmapping any mapped sections which straddle between the old length and the new. Any sections lost this way must be remapped by you if you want them to (this could not be done automatically because the mapping address which you may be relying upon could change). There is a difference between writing off the end of the file and truncate()-ing it - only the latter extends the ability to map the new data.

The mapOffset() method lets you see if any arbitrary file ptr offset maps into the currently mapped sections. It returns the address of the corresponding location in memory or zero if that section is not mapped.

Like all file type i/o classes, QMemMap can perform automatic CR/LF translation as well as UTF-8 to UTF-16 and UTF-32 conversion. If you enable IO_Translate, the file data is probed and its unicode type determined such that the file unicode type is transparently converted into UTF-8 and back into its original form. This allows your code to work exclusively in UTF-8 using the standard FX::FXString functions. You can set the type of output using setUnicodeTranslation(). Note that for obvious reasons if you access the data directly using mapIn(), no translation is performed.

Note:

There is a bug on WinNT whereby if you map the same file in for write access using two or more QMemMap's, the second and thereafter will never allow any mapping whatsoever. Either share the one QMemMap, or accept lower performance. This problem does not affect multiple read access instances or a write with many reads - only two or more write access instances.

Another misfeature on WinNT is, when under Terminal Services or the second or later non-administrator user to log on to a WinXP machine, the SeCreateGlobalPrivilege is not given - this means TnFOX can no longer create shared memory that all processes can see. In this situation, TnFOX tries to open the global version of the shared memory name which if it fails, it then creates it in the local namespace. This behaviour can be unexpected especially as to most users of WinXP one login session appears identical to a normal one.

Definition at line 165 of file QMemMap.h.

Public Types
	File
	Memory
enum	Type { File, Memory }
typedef FXfval	Offset
	Default
	Unix
	MacOS
	MSDOS
	NoTranslation
	UTF8
	UTF16
	UTF16LE
	UTF32
	UTF32LE
enum	CRLFType { Default, Unix, MacOS, MSDOS }
enum	UnicodeType {   NoTranslation, UTF8, UTF16, UTF16LE,   UTF32, UTF32LE }
Public Member Functions
	QMemMap ()
	QMemMap (const FXString &filename)
	QMemMap (QFile &file)
	QMemMap (const FXString &name, FXuval len)
	~QMemMap ()
const FXString &	name () const
void	setName (const FXString &name)
bool	isUnique () const
void	setUnique (bool v)
Type	type () const
void	setType (Type type)
bool	exists () const
bool	remove ()
FXfval	reloadSize ()
FXfval	mappableSize () const
void	maximiseMappableSize ()
void *	mapIn (FXfval offset=0, FXfval amount=(FXfval)-1, bool copyOnWrite=false)
void	mapOut (FXfval offset=0, FXfval amount=(FXfval)-1)
void	mapOut (void *area)
bool	mappedRegion (MappedRegion *current, MappedRegion *next=0, FXfval offset=(FXfval)-1) const
void *	mapOffset (FXfval offset=(FXfval)-1) const
virtual bool	open (FXuint mode)
virtual void	close ()
virtual void	flush ()
virtual FXfval	size () const
virtual void	truncate (FXfval size)
virtual FXfval	at () const
virtual bool	at (FXfval newpos)
virtual bool	atEnd () const
virtual const FXACL &	permissions () const
virtual void	setPermissions (const FXACL &perms)
virtual FXuval	readBlock (char *data, FXuval maxlen)
virtual FXuval	writeBlock (const char *data, FXuval maxlen)
virtual FXuval	readBlockFrom (char *data, FXuval maxlen, FXfval pos)
virtual FXuval	writeBlockTo (FXfval pos, const char *data, FXuval maxlen)
virtual int	getch ()
virtual int	putch (int c)
virtual int	ungetch (int c)
FXuint	flags () const
FXuint	mode () const
FXuint	state () const
CRLFType	crlfFormat () const
void	setCRLFFormat (CRLFType type)
UnicodeType	unicodeTranslation () const
void	setUnicodeTranslation (UnicodeType type)
bool	isBuffered () const
bool	isRaw () const
bool	isTranslated () const
bool	isUTF16Translated () const
bool	isUTF32Translated () const
bool	isReadable () const
bool	isWriteable () const
bool	isWritable () const
bool	isReadWrite () const
bool	isClosed () const
bool	isInactive () const
bool	isOpen () const
virtual bool	isSynchronous () const
FXuval	readBlock (FXuchar *data, FXuval maxlen)
FXuval	writeBlock (const FXuchar *data, FXuval maxlen)
virtual FXuval	readLine (char *data, FXuval maxlen)
FXuval	readBlockFrom (FXuchar *data, FXuval maxlen, FXfval pos)
FXuval	writeBlockTo (FXfval pos, const FXuchar *data, FXuval maxlen)
FXfval	shredData (FXfval offset, FXfval len=(FXfval)-1)
Static Public Member Functions
static FXACL	permissions (const FXString &name)
static void	setPermissions (const FXString &name, const FXACL &perms)
static UnicodeType	determineUnicodeType (FXuchar *data, FXuval len) throw ()
static FXuval	applyCRLF (FXuchar *FXRESTRICT output, const FXuchar *FXRESTRICT input, FXuval outputlen, FXuval &inputlen, CRLFType crlftype=Default, UnicodeType utftype=NoTranslation)
static FXuval	removeCRLF (FXuchar *FXRESTRICT output, const FXuchar *FXRESTRICT input, FXuval outputlen, FXuval &inputlen, UnicodeType utftype=NoTranslation)
Protected Member Functions
void	setFlags (int f)
void	setMode (int m)
void	setState (int s)
Protected Attributes
FXfval	ioIndex
Friends
FXAPI FXStream &	operator<< (FXStream &s, QIODevice &i)
FXAPI FXStream &	operator>> (FXStream &s, QIODevice &i)
Classes
struct	MappedRegion
	A mapped region within the map. More...

---

The documentation for this class was generated from the following file:

• QMemMap.h

---

(C) 2002-2009 Niall Douglas. Some parts (C) to assorted authors.