FX::QMemMap Class Reference
[File type i/o devices]

#include <QMemMap.h>

Inheritance diagram for FX::QMemMap:

Detailed Description

An i/o device accessing mapped memory.

Mapped memory is conceptually a portion of a disc-based file indirectly accessed as though it were a region of memory. All modern operating systems support this facility, yet astonishingly little use is made of it in most computer programs. Quite possibly this is because it is harder to use than it should be which is a shame because it makes available the very best file access caching and buffering to your program completely transparently.

QMemMap addresses this by providing transparent usage of mapped memory and portably too. There are no issues with accidentally running off the end of the mapped section nor off the start. Use as shared memory between processes is also ridiculously easy - you simply name the device the same in both processes and voil�, you're now working with the same patch of memory! Remember to synchronise multiple process access - the most portable is via msgs using a pipe, however if there is low contention in access it can be done via FX::QShrdMemMutex.

You can map in an existing FX::QFile device and indeed internally QMemMap creates one if you don't specify a FX::QFile but do a filename. You should note that as a result, QMemMap can lose track of the real file length under the same conditions as FX::QFile for which reloadSize() is also provided. If you choose shared memory (QMemMap::Memory) rather than a filename (QMemMap::File) then the file used is kept in memory as much as possible rather than being flushed to disc as soon as possible.

If you wish to convert a FX::QBuffer into a QMemMap, simply create an FX::FXStream refering to the QMemMap and do

FXStream ds((QMemMap) x);
ds << (QBuffer) y;

QMemMap is heavily optimised including custom getch() and putch() especially for mostly sequential transfers (where at() is not moving more than FXProcess::pageSize() at a time). Also file changes moving forwards are better than backwards. In ideal conditions, not much more than a memcpy() is being performed so the bigger the chunks you give it, the better.

Default security is for FX::FXACLEntity::everything() to have full access when the type is mapped memory - file maps deny everything access except the local process. This makes sense as mapped memory is generally for inter-process communication - if however it's a private region for communication with a known process it makes sense to FXRESTRICT access to the user running that process at least. Note that until the map is opened, permissions() returns what will be applied to the map on open() rather than the map itself - if you want the latter, use the static method. Note also that the map's security is not linked in any way with that of any file underlying it - you must set them separately. Note that on POSIX, the underlying file's permissions are those for the map and setting them here has no effect - however, if it's shared memory then the permissions do have an effect.

Like FX::QPipe, the shared memory name is deleted by its creator on POSIX only (on Windows it lasts until the last thing referring to it closes). If you don't want this, specify IO_DontUnlink in the flags to open().

Usage:

Simply instantiate, set filename or name and open(). Once open, use mapIn() to map in a specified section and mapOut() to map it out. On 32 bit architectures, files bigger than ~1Gb on some operating systems will fail to map in so be aware of the consequences of the defaults in mapIn() which map everything in. Failures to map in due to address space exhaustion cause mapIn() to return zero rather than an exception - however, the mapping remains on the books and may get mapped in silently the next time something is mapped out. See FX::FXProcess::virtualAddrSpaceLeft().

This is done this way because the other important facility QMemMap offers is transparently using the mapped section(s) when the file pointer is within one and doing normal readBlock() and writeBlock() with the FX::QFile when outside. Thus you can afford to map in the most commonly used portion(s) of your file and let the rest happen via buffered i/o as usual.

Warning:: You really do not want to read or write parts of a file which have been mapped in using FX::QFile directly. This inevitably causes data corruption. To avoid this, open() calls flush() on its FX::QFile if it's already open so therefore after this point, all i/o on the file should be done via QMemMap.

Indeed unless you need absolutely critical sustained sequential reads or writes, I'd recommend doing all your file i/o via FX::QMemMap instead of via FX::QFile and indeed I've made the non static methods exactly the same. Memory mapped files are demand-read often with intelligent prefetching as read accesses are performed from its mapped section. Writes almost always remain in the RAM backing the mapped section until the system is idle (ie; not doing more reads) so you get the maximum performance from your hardware. Even better when free memory is tight, the operating system will write out dirty pages sooner rather than later thus getting the best of all possible worlds. On my development system running Win2k I get a three-fold speed increase copying one file to another over QFile and that's even with NTFS's excellent caching of buffered i/o.

Note:: I've found that memory mapped file i/o is considerably faster on Linux when using ReiserFS rather than ext3.

Extending the file size with truncate() works as expected - however depending on the host OS, it can require recreating the map which implies deletion and recreation of all mapped sections (which may move position). Shortening the file requires wholly unmapping any mapped sections which straddle between the old length and the new. Any sections lost this way must be remapped by you if you want them to (this could not be done automatically because the mapping address which you may be relying upon could change). There is a difference between writing off the end of the file and truncate()-ing it - only the latter extends the ability to map the new data.

The mapOffset() method lets you see if any arbitrary file ptr offset maps into the currently mapped sections. It returns the address of the corresponding location in memory or zero if that section is not mapped.

Like all file type i/o classes, QMemMap can perform automatic CR/LF translation as well as UTF-8 to UTF-16 and UTF-32 conversion. If you enable IO_Translate, the file data is probed and its unicode type determined such that the file unicode type is transparently converted into UTF-8 and back into its original form. This allows your code to work exclusively in UTF-8 using the standard FX::FXString functions. You can set the type of output using setUnicodeTranslation(). Note that for obvious reasons if you access the data directly using mapIn(), no translation is performed.

Note:: There is a bug on WinNT whereby if you map the same file in for write access using two or more QMemMap's, the second and thereafter will never allow any mapping whatsoever. Either share the one QMemMap, or accept lower performance. This problem does not affect multiple read access instances or a write with many reads - only two or more write access instances.
Another misfeature on WinNT is, when under Terminal Services or the second or later non-administrator user to log on to a WinXP machine, the SeCreateGlobalPrivilege is not given - this means TnFOX can no longer create shared memory that all processes can see. In this situation, TnFOX tries to open the global version of the shared memory name which if it fails, it then creates it in the local namespace. This behaviour can be unexpected especially as to most users of WinXP one login session appears identical to a normal one.


Public Types
enum	Type { File, Memory }
enum	CRLFType { Default, Unix, MacOS, MSDOS }
enum	UnicodeType { NoTranslation, UTF8, UTF16, UTF16LE, UTF32, UTF32LE }
typedef FXfval	Offset
Public Member Functions
	QMemMap (const FXString &filename)
	QMemMap (QFile &file)
	QMemMap (const FXString &name, FXuval len)
const FXString &	name () const
void	setName (const FXString &name)
bool	isUnique () const
void	setUnique (bool v)
Type	type () const
void	setType (Type type)
bool	exists () const
bool	remove ()
FXfval	reloadSize ()
FXfval	mappableSize () const
void	maximiseMappableSize ()
void *	mapIn (FXfval offset=0, FXfval amount=(FXfval)-1, bool copyOnWrite=false)
void	mapOut (FXfval offset=0, FXfval amount=(FXfval)-1)
void	mapOut (void *area)
bool	mappedRegion (MappedRegion current, MappedRegion next=0, FXfval offset=(FXfval)-1) const
void *	mapOffset (FXfval offset=(FXfval)-1) const
virtual bool	open (FXuint mode)
virtual void	close ()
virtual void	flush ()
virtual FXfval	size () const
virtual void	truncate (FXfval size)
virtual FXfval	at () const
virtual bool	at (FXfval newpos)
virtual bool	atEnd () const
virtual const FXACL &	permissions () const
virtual void	setPermissions (const FXACL &perms)
virtual FXuval	readBlock (char *data, FXuval maxlen)
virtual FXuval	writeBlock (const char *data, FXuval maxlen)
virtual FXuval	readBlockFrom (char *data, FXuval maxlen, FXfval pos)
virtual FXuval	writeBlockTo (FXfval pos, const char *data, FXuval maxlen)
virtual int	getch ()
virtual int	putch (int c)
virtual int	ungetch (int c)
FXuint	flags () const
FXuint	mode () const
FXuint	state () const
CRLFType	crlfFormat () const
void	setCRLFFormat (CRLFType type)
UnicodeType	unicodeTranslation () const
void	setUnicodeTranslation (UnicodeType type)
bool	isBuffered () const
bool	isRaw () const
bool	isTranslated () const
bool	isUTF16Translated () const
bool	isUTF32Translated () const
bool	isReadable () const
bool	isWriteable () const
bool	isWritable () const
bool	isReadWrite () const
bool	isClosed () const
bool	isInactive () const
bool	isOpen () const
virtual bool	isSynchronous () const
FXuval	readBlock (FXuchar *data, FXuval maxlen)
FXuval	writeBlock (const FXuchar *data, FXuval maxlen)
virtual FXuval	readLine (char *data, FXuval maxlen)
FXuval	readBlockFrom (FXuchar *data, FXuval maxlen, FXfval pos)
FXuval	writeBlockTo (FXfval pos, const FXuchar *data, FXuval maxlen)
FXfval	shredData (FXfval offset, FXfval len=(FXfval)-1)
Static Public Member Functions
static FXACL	permissions (const FXString &name)
static void	setPermissions (const FXString &name, const FXACL &perms)
static UnicodeType	determineUnicodeType (FXuchar *data, FXuval len) throw ()
static FXuval	applyCRLF (FXuchar FXRESTRICT output, const FXuchar FXRESTRICT input, FXuval outputlen, FXuval &inputlen, CRLFType crlftype=Default, UnicodeType utftype=NoTranslation)
static FXuval	removeCRLF (FXuchar FXRESTRICT output, const FXuchar FXRESTRICT input, FXuval outputlen, FXuval &inputlen, UnicodeType utftype=NoTranslation)
Protected Member Functions
void	setFlags (int f)
void	setMode (int m)
void	setState (int s)
Protected Attributes
FXfval	ioIndex
Friends
FXAPI FXStream &	operator<< (FXStream &s, QIODevice &i)
FXAPI FXStream &	operator>> (FXStream &s, QIODevice &i)
Classes
struct	MappedRegion
	A mapped region within the map. More...

Member Enumeration Documentation

enum FX::QMemMap::Type

The types of mapped memory there are.

Enumerator:

File	The name refers to a file based on disc.
Memory	The name refers to a name of shared memory.

enum FX::QIODevice::CRLFType [inherited]

The type of CR/LF encoding you want.

Enumerator:

Default	Uses the host OS format.
Unix	Uses ASCII 10 to delimit lines.
MacOS	Uses ASCII 13 to delimit lines.
MSDOS	Uses ASCII 13,10 to delimit lines.

enum FX::QIODevice::UnicodeType [inherited]

The type of unicode translation you want.

Enumerator:

NoTranslation	Performs no translation.
UTF8	Outputs in native UTF-8.
UTF16	Converts between native (UTF-8) and UTF-16 big endian.
UTF16LE	Converts between native (UTF-8) and UTF-16 little endian.
UTF32	Converts between native (UTF-8) and UTF-32 big endian.
UTF32LE	Converts between native (UTF-8) and UTF-32 little endian.

Constructor & Destructor Documentation

FX::QMemMap::QMemMap ( const FXString & filename )

Constructs an instance operating on file filename.

FX::QMemMap::QMemMap ( QFile & file )

Constructs an instance using an external FX::QFile.

FX::QMemMap::QMemMap	(	const FXString &	name,
		FXuval	len
	)

Constructs an instance working with named shared memory of length len. If name is empty, sets to be unique.

Member Function Documentation

const FXString& FX::QMemMap::name ( ) const

Returns the filename or shared memory name being addressed by this device.

void FX::QMemMap::setName ( const FXString & name )

Sets the filename or shared memory name being addressed by this device. Closes the old file first.

bool FX::QMemMap::isUnique ( ) const

Returns true if the shared memory name is unique.

void FX::QMemMap::setUnique ( bool v )

Sets if the shared memory name is unique.

Type FX::QMemMap::type ( ) const

Returns the type of the memory map.

void FX::QMemMap::setType ( Type type )

Sets whether to interpret the name as a filename or shared memory.

bool FX::QMemMap::exists ( ) const

Returns true if the file name points to an existing file.

bool FX::QMemMap::remove ( )

Deletes the file name, closing the file first if open. Returns false if file doesn't exist.

FXfval FX::QMemMap::reloadSize ( )

Reloads the size of the file. See description above.

FXfval FX::QMemMap::mappableSize ( ) const

Returns the current mappable extent of the file (which may be shorter than the file length).

void FX::QMemMap::maximiseMappableSize ( )

Extends the mappable extent of the file to the current file length.

void* FX::QMemMap::mapIn	(	FXfval	offset = `0`,
		FXfval	amount = `(FXfval)-1`,
		bool	copyOnWrite = `false`
	)

Returns:: Where in memory the section was mapped to. Zero if could not be mapped (eg; ran out of address space which is common on 32 bit architectures).

Parameters:

	offset	From where in the file to map (should be zero for shared memory). Must be a multiple of FXProcess::pageSize().
	amount	How much to map in. (FXfval) -1 maps in the length of the file (bear in mind mapping in large files can exhaust available address space).
	copyOnWrite	True if writes to the map should not be written to the file. Effectively disables shared memory.

Maps the specified section of the file into memory. If there is insufficient address space free to fit it all in, fails silently and returns zero. Note that if this map overlaps any existing map, that map is removed first.

void FX::QMemMap::mapOut	(	FXfval	offset = `0`,
		FXfval	amount = `(FXfval)-1`
	)

Maps out the specified section. If this cross-sects an existing section(s), those sections are mapped out in their entirity (because remaps may move location).

void FX::QMemMap::mapOut ( void * area )

This is an overloaded member function, provided for convenience. It differs from the above function only in what argument(s) it accepts. Maps out a section based on the address returned from a previous mapIn().

bool FX::QMemMap::mappedRegion	(	MappedRegion *	current,
		MappedRegion *	next = `0`,
		FXfval	offset = `(FXfval)-1`
	)			const

Returns a description of the mapped region containing the specified offset plus that of the region following, returning zero in all members if that offset is not within a region which has been mapped. The default offset is the current file pointer. Returns false if no region exists between the specified offset and the end of the file.

void* FX::QMemMap::mapOffset ( FXfval offset = (FXfval)-1 ) const

Returns the address within memory of where the specified offset into the file is mapped (which by default is the current file pointer). Returns zero if that place is not mapped into memory.

virtual bool FX::QMemMap::open ( FXuint mode ) [virtual]

Opens the device for the specified access.

Implements FX::QIODevice.

virtual void FX::QMemMap::close ( ) [virtual]

Closes the device.

Implements FX::QIODevice.

virtual void FX::QMemMap::flush ( ) [virtual]

Flushes all open mapped sections to disc, then any underlying file.

Implements FX::QIODevice.

virtual FXfval FX::QMemMap::size ( ) const [virtual]

Returns the size of the data being accessed by the device.

Implements FX::QIODevice.

virtual void FX::QMemMap::truncate ( FXfval size ) [virtual]

Sets a new length of file, unmapping any sections containing the part being removed

Note:: This is not especially a fast call as the mapped sections must be recreated. On the other hand, it is the only way to make newly written data off the end mappable.

Implements FX::QIODevice.

virtual FXfval FX::QMemMap::at ( ) const [virtual]

Returns the current file pointer within the device

Note:: With IO_Translate enabled, this increments faster than the return values from either readBlock() or writeBlock()

Reimplemented from FX::QIODevice.

virtual bool FX::QMemMap::at ( FXfval newpos ) [virtual]

Sets the current file pointer

Note:: With IO_Translate enabled, setting to anything other than a previously read at() can be hazardous.

Reimplemented from FX::QIODevice.

virtual bool FX::QMemMap::atEnd ( ) const [virtual]

Returns true if there is no more data available to be read from the device.

Reimplemented from FX::QIODevice.

virtual const FXACL& FX::QMemMap::permissions ( ) const [virtual]

Returns the ACL for this device.

Reimplemented from FX::QIODevice.

virtual void FX::QMemMap::setPermissions ( const FXACL & ) [virtual]

Sets the ACL for this device.

Reimplemented from FX::QIODevice.

static FXACL FX::QMemMap::permissions ( const FXString & name ) [static]

Returns the permissions for the memory mapped section called name.

static void FX::QMemMap::setPermissions	(	const FXString &	name,
		const FXACL &	perms
	)			`[static]`

Sets the permissions for the memory mapped section called name.

virtual FXuval FX::QMemMap::readBlock	(	char *	data,
		FXuval	maxlen
	)			`[virtual]`

Reads up to the specified quantity of bytes into the buffer, returning how much was actually read

Note:: With IO_Translate enabled, this routine regularly returns less read than maxlen (as the CR's are stripped out).

Implements FX::QIODevice.

virtual FXuval FX::QMemMap::writeBlock	(	const char *	data,
		FXuval	maxlen
	)			`[virtual]`

Writes up to the specified quantity of bytes from the buffer, returning how much was actually written. Note that less being written due to error is returned as an exception, but some devices may write less in a non-error situation.

Implements FX::QIODevice.

virtual FXuval FX::QMemMap::readBlockFrom	(	char *	data,
		FXuval	maxlen,
		FXfval	pos
	)			`[virtual]`

Combines an at() and readBlock() together. Can be much more efficient than those two operations individually with some i/o devices, plus it's synchronous and thus threadsafe

Implements FX::QIODevice.

virtual FXuval FX::QMemMap::writeBlockTo	(	FXfval	pos,
		const char *	data,
		FXuval	maxlen
	)			`[virtual]`

Combines an at() and writeBlock() together. Can be much more efficient than those two operations individually with some i/o devices, plus it's synchronous and thus threadsafe

Implements FX::QIODevice.

virtual int FX::QMemMap::getch ( ) [virtual]

Reads a single byte. Returns -1 for no data found.

Reimplemented from FX::QIODevice.

virtual int FX::QMemMap::putch ( int c ) [virtual]

Writes a single byte.

Reimplemented from FX::QIODevice.

virtual int FX::QMemMap::ungetch ( int c ) [virtual]

Pushes back a byte to the read buffer.

Implements FX::QIODevice.

FXuint FX::QIODevice::flags ( ) const [inline, inherited]

Returns the flags of this device.

FXuint FX::QIODevice::mode ( ) const [inline, inherited]

Returns the mode of this device

See also:: QIODeviceOpenFlags

FXuint FX::QIODevice::state ( ) const [inline, inherited]

Returns the state of this device

See also:: QIODeviceStateFlags

CRLFType FX::QIODevice::crlfFormat ( ) const [inline, inherited]

Returns the CR/LF format of this device.

void FX::QIODevice::setCRLFFormat ( CRLFType type ) [inline, inherited]

Sets the CR/LF format for output of this device.

UnicodeType FX::QIODevice::unicodeTranslation ( ) const [inline, inherited]

Returns the unicode translation of this device.

void FX::QIODevice::setUnicodeTranslation ( UnicodeType type ) [inline, inherited]

Sets the unicode translation of this device.

bool FX::QIODevice::isBuffered ( ) const [inline, inherited]

Returns true if the device is buffered.

bool FX::QIODevice::isRaw ( ) const [inline, inherited]

Returns true if the device is unbuffered.

bool FX::QIODevice::isTranslated ( ) const [inline, inherited]

Returns true if the device is LR/CF translated.

bool FX::QIODevice::isUTF16Translated ( ) const [inline, inherited]

Returns true if the device is UTF-16 translated.

bool FX::QIODevice::isUTF32Translated ( ) const [inline, inherited]

Returns true if the device is UTF-32 translated.

bool FX::QIODevice::isReadable ( ) const [inline, inherited]

Returns true if the device is readable.

bool FX::QIODevice::isWriteable ( ) const [inline, inherited]

Returns true if the device is writeable.

bool FX::QIODevice::isReadWrite ( ) const [inline, inherited]

Returns true if the device is readable & writeable.

bool FX::QIODevice::isClosed ( ) const [inline, inherited]

Returns true if the device is closed.

bool FX::QIODevice::isOpen ( ) const [inline, inherited]

Returns true if the device is opened.

virtual bool FX::QIODevice::isSynchronous ( ) const [inline, virtual, inherited]

Returns true if this device is a synchronous device.

Reimplemented in FX::QIODeviceS, and FX::QSSLDevice.

virtual FXuval FX::QIODevice::readLine	(	char *	data,
		FXuval	maxlen
	)			`[virtual, inherited]`

Reads data until an end-of-line or maxlen is exceeded.

static UnicodeType FX::QIODevice::determineUnicodeType	(	FXuchar *	data,
		FXuval	len
	)			throw () `[static, inherited]`

Looks at a sample of data and determines what kind of Unicode text it is, returning UnicodeType::NoTranslation if it isn't text. This routine isn't foolproof, but it's a good guess

static FXuval FX::QIODevice::applyCRLF	(	FXuchar *FXRESTRICT	output,
		const FXuchar *FXRESTRICT	input,
		FXuval	outputlen,
		FXuval &	inputlen,
		CRLFType	crlftype = `Default`,
		UnicodeType	utftype = `NoTranslation`
	)			`[static, inherited]`

Applies CR/LF and optional UTF-x translation returning bytes output. If outputlen would run out before inputlen can be exhausted it will return early (eg; if mid newline, or mid UTF-x sequence). You should rewind processing to the difference between entrant inputlen and returned inputlen.

static FXuval FX::QIODevice::removeCRLF	(	FXuchar *FXRESTRICT	output,
		const FXuchar *FXRESTRICT	input,
		FXuval	outputlen,
		FXuval &	inputlen,
		UnicodeType	utftype = `NoTranslation`
	)			`[static, inherited]`

Removes CR/LF translation intelligently (ie; self-adjusts to MS-DOS, Unix and MacOS formats or any mixture of these) and can perform optional UTF-x translation, returning bytes output. If outputlen would run out before inputlen can be exhausted it will return early (eg; if mid newline, or mid UTF-x sequence). You should rewind processing to the difference between entrant inputlen and returned inputlen.

FXfval FX::QIODevice::shredData	(	FXfval	offset,
		FXfval	len = `(FXfval)-1`
	)			`[inherited]`

Destroys the len bytes of data from offset offset into the file. Restores the file pointer afterwards and returns how much data was shredded before end of file if encountered. You must have the device open for both reading and writing for this call to succeed.

void FX::QIODevice::setFlags ( int f ) [inline, protected, inherited]

Sets the flags.

void FX::QIODevice::setMode ( int m ) [inline, protected, inherited]

Sets the mode.

void FX::QIODevice::setState ( int s ) [inline, protected, inherited]

Sets the state.

Friends And Related Function Documentation

FXAPI FXStream& operator<<	(	FXStream &	s,
		QIODevice &	i
	)			`[friend, inherited]`

Appends the contents of an i/o device to stream s

Warning:: This operation is not thread-safe

FXAPI FXStream& operator>>	(	FXStream &	s,
		QIODevice &	i
	)			`[friend, inherited]`

Reads all available contents of the stream s to an i/o device, replacing its current contents and resetting the file pointer to the start

Warning:: This operation is not thread-safe

The documentation for this class was generated from the following file:

QMemMap.h

FX::QMemMap Class Reference [File type i/o devices]

Detailed Description

Usage:

Public Types

Public Member Functions

Static Public Member Functions

Protected Member Functions

Protected Attributes

Friends

Classes

Member Enumeration Documentation

Constructor & Destructor Documentation

Member Function Documentation

Friends And Related Function Documentation

FX::QMemMap Class Reference
[File type i/o devices]