#
##
## SPDX-FileCopyrightText: © 2007-2021 Benedict Verhegghe <bverheg@gmail.com>
## SPDX-License-Identifier: GPL-3.0-or-later
##
## This file is part of pyFormex 2.6 (Mon Aug 23 15:13:50 CEST 2021)
## pyFormex is a tool for generating, manipulating and transforming 3D
## geometrical models by sequences of mathematical operations.
## Home page: https://pyformex.org
## Project page: https://savannah.nongnu.org/projects/pyformex/
## Development: https://gitlab.com/bverheg/pyformex
## Distributed under the GNU General Public License version 3 or later.
##
## This program is free software: you can redistribute it and/or modify
## it under the terms of the GNU General Public License as published by
## the Free Software Foundation, either version 3 of the License, or
## (at your option) any later version.
##
## This program is distributed in the hope that it will be useful,
## but WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
## GNU General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with this program. If not, see http://www.gnu.org/licenses/.
##
"""A new format for saving pyFormex geometry/projects to file.
This module provides some functions to save pyFormex geometry and
other objects to a file, and load them back from these files.
For this purpose we use a new, experimental file format and the
filename extension .PZF (pyFormex zip format).
The new PZF format is experimental, and not complete (yet). Do not use it
yet to store data that have to persist over a long period. One day however
it could replace the current .PGF (geometry) and/or PYF (project) formats.
The new format is very robust, is easy to implement and extend, provides
easy ways of upgrading without losing contents, and guarantees openness to
other softwares and portability to other OSes and architectures.
It currently stores Formex, Mesh anf TriSurface objects, togerther with
the Fields added to the object, and even the drawing attributes.
Basically, PZF files are zip archives written with NumPy's savez function.
Being a zip file, they can be opened with most modern file managers, allowing
the user a view on what's inside. In future we could even provide facilities
to extract selected objects from the file and to open an archive directly
in pyFormex by just double clicking on the filename.
It also provides possibility to compress and even password-protect the
PZF contents.
Since most information in pyFormex objects is stored in NumPy arrays,
the choice of numpy.savez to write a whole set of arrays to a zip file
saved us a lot of own development, and guarantees future support and
stability of the format. However, numpy.savez only saves arrays, and we
need to store more things for pyFormex objects: the class name, some non-array
attributes (usually strings) like 'eltype', and the 'fields' and 'attrib'
attibutes. Fields are again mostly an array plus some strings, but 'attrib'
is a dict that could be quite complex, although it usually isn't.
And the complex things usually can be avoided: instead of setting
'color=some_large_color_array' in the attributes, one can add
'some_large_color_array' as a Field to the object, with 'some_field_name',
and then set a drawing attribute 'color=fld:some_field_name'.
Being a zip archive, the contents of it are individual files. Every
file contains exactly one numpy array. We encode some of the string data
into the file names: the object name, its class name, the attribute name.
For a field we also encode the field name and field type. For example,
the contents of the 'saveload.pzf' file in the pyformex/data folder is::
F__Formex__coords.npy
F__Formex__prop.npy
M__Mesh__attrib.npy
M__Mesh__coords.npy
M__Mesh__elems.npy
M__Mesh__eltype__quad4.npy
M__Mesh__field__elemc__dist3c.npy
M__Mesh__field__node__dist.npy
M__Mesh__field__node__dist3n.npy
T__TriSurface__attrib.npy
T__TriSurface__coords.npy
T__TriSurface__elems.npy
spiral__PolyLine__attrib.npy
spiral__PolyLine__coords.npy
The .npy extension of the files stresses the fact that each individual file
stores a numpy array in .npy format. They could be restored to plain numpy
arrays, but the pyFormex reader will take info from the file names and combine
them into proper pyFormex objects. The file name is made up of different
fields, separated by two underscores '__'.
The first two fields are the object's name and class. Evidently, you can
not use a double underscore in an object name. From the file names, it is
immediately clear what this archive contains: a Formex 'F', a Mesh 'M' and
a TriSurface 'T'. The file manager will normally show the file names sorted
so that it is easy to see what each object consists of. It is also conceivable
to delete some files from the archive, or to extract the archive, move, delete,
change or rename some files, and zip the resulting files again.
The third field in the file name is the class attribute which has its data
stored in the file. For attributes like 'coords', 'elems' 'prop', that are
numpy arrays, this is evident. Attributes whose value is a string are a bit
more complex: if the string is short and simple (does not contain strange
control or unicode characters), it can be encoded directly in the file name
as the fourth field: the Mesh 'eltype' field is an example.
If the string is long or contains strange characters, it is stored as a
charcter array (NEW: we can now store directly as text file. See below)
This needs special attention on readback to allow portability over Python2
and Python3. The 'attrib' attribute is even more
complex: it is a dict that may contain different type of objects. The 'color'
key for example could have a string or an int or float array type. Currently
we are serializing the whole 'attrib' in a json format string, and store the
result (as a char array).
If the third component of the file name is 'field', it obviously stores a
Field of the object. This attribute can occur multiple times and has two
more components in its name: the Field's fldtype and fldname. The contents
of the file is again an array with the Field data.
The practical use is by these two function::
from pyformex.saveload import savePZF, loadPZF
savePZF(filename,**kargs)
dic = loadPZF(filename)
Though it is currently not enforced, we suggest to always use an extension
'.pzf' for the file name. The dictionary ``dic`` contains the pyFormex objects
with their names as keys.
Data of string type can now be stored directly as a text file in the zip
archive. This is especially convenient for things that could allow editing.
Modern file managers allow transparent unpacking of a file in the zip archive,
editing of the file and store back into the archive.
Limitations: currently, only objects of the following classes can be stored:
Coords, Formex, Mesh, TriSurface, PolyLine, BezierSpline, CoordSys,
Camera settings.
See also: example SaveLoad
"""
import sys
import json
import zipfile
import numpy as np
import pyformex as pf
from pyformex import utils
from pyformex.path import Path
from pyformex.coords import Coords
from pyformex.geometry import Geometry
from pyformex.formex import Formex
from pyformex.mesh import Mesh
from pyformex.trisurface import TriSurface
from pyformex.plugins.curve import PolyLine, BezierSpline
from pyformex.plugins.nurbs import NurbsCurve, NurbsSurface
from pyformex.coordsys import CoordSys
from pyformex.config import Config
_pzf_version = '1.0'
__all__ = ['savePZF', 'loadPZF', 'listPZF', 'listPZFobj']
[docs]def dict2Config(d):
"""Convert a dict to a Config.
A Config is a specialized dict type that can only take some
kind of values. This will raise an exception if other values
are in the dict.
"""
C = Config()
for k in d:
v = d[k]
if isinstance(v, (str, int, np.integer, float, np.floating, tuple, list)):
C[k] = v
elif isinstance(v, np.ndarray):
C[k] = v.tolist()
else:
raise ValueError("Can not store key '%s' of type '%s' in Config." % (k, type(v)))
return C
[docs]def bytes2str(b):
"""Convert bytes to str"""
return str(b)[2:-1]
[docs]def readStr(filename):
"""Read a string from a saved numpy character
The character array may be of kind 'S' (bytes) or 'U' (unicode)
but always returns an object of class str.
"""
a = np.load(filename)
if a.dtype.kind == 'S':
a = a.astype('U')
return str(a)
def Coords_pzf_dict(self):
return {'data': self}
[docs]def Geometry_pzf_dict(self):
"""Construct common part of all Geometry pzf dicts"""
kargs = {}
kargs['coords'] = self.coords
if self.prop is not None:
kargs['prop'] = self.prop
if self.fields:
for k in self.fields:
f = self.fields[k]
kargs['field__%s__%s' % (f.fldtype, k)] = f.data
if self.attrib.keys():
kargs['attrib'] = json.dumps(self.attrib)
return kargs
def Formex_pzf_dict(self):
kargs = Geometry.pzf_dict(self)
if self.eltype:
kargs["%s__%s" % ('eltype', self.eltype)] = []
return kargs
def Mesh_pzf_dict(self):
kargs = Geometry.pzf_dict(self)
kargs['elems'] = self.elems
kargs["%s__%s" % ('eltype', self.elName())] = []
return kargs
def TriSurface_pzf_dict(self):
kargs = Geometry.pzf_dict(self)
kargs['elems'] = self.elems
return kargs
def PolyLine_pzf_dict(self):
kargs = Geometry.pzf_dict(self)
if self.closed:
kargs['closed'] = []
return kargs
def BezierSpline_pzf_dict(self):
kargs = Geometry.pzf_dict(self)
kargs['control'] = kargs['coords']
kargs['degree__%s' % self.degree] = []
del kargs['coords']
if self.closed:
kargs['closed'] = []
return kargs
def NurbsCurve_pzf_dict(self):
kargs = {
'control': self.coords,
'knots': self.knotu.values(),
'degree__%s' % self.degree: [],
}
if self.closed:
kargs['closed'] = []
return kargs
def NurbsSurface_pzf_dict(self):
kargs = {
'control': self.coords,
'knotu': self.knotu.values(),
'knotv': self.knotv.values(),
'dict': Config({
'degree': self.degree,
'closed': self.closed,
})
}
return kargs
def CoordSys_pzf_dict(self):
return {
'rot': self.rot,
'trl': self.trl
}
def Config_pzf_dict(self):
return {'settings': str(self)}
# Install the above in their classes
Coords.pzf_dict = Coords_pzf_dict
Geometry.pzf_dict = Geometry_pzf_dict
Formex.pzf_dict = Formex_pzf_dict
Mesh.pzf_dict = Mesh_pzf_dict
TriSurface.pzf_dict = TriSurface_pzf_dict
PolyLine.pzf_dict = PolyLine_pzf_dict
BezierSpline.pzf_dict = BezierSpline_pzf_dict
NurbsCurve.pzf_dict = NurbsCurve_pzf_dict
NurbsSurface.pzf_dict = NurbsSurface_pzf_dict
CoordSys.pzf_dict = CoordSys_pzf_dict
Config.pzf_dict = Config_pzf_dict
# Class initializations requiring positional arguments need to be declared here
Formex.pzf_args = ['coords']
TriSurface.pzf_args = ['coords', 'elems']
[docs]def save2zip(zipf, namedict):
"""Save a dict to an open zip file
Parameters
----------
namedict: dict
Dict with objects to store. The keys should be valid Python
variable names.
"""
from numpy.lib import format
if sys.version_info >= (3, 6):
# Since Python 3.6 it is possible to write directly to a ZIP file.
def write_text(val, fname, zipf):
force_zip64 = len(val) >= 2**30
with zipf.open(fname, 'w', force_zip64=force_zip64) as fil:
fil.write(val)
def write_array(val, fname, zipf):
force_zip64 = val.nbytes >= 2**30
with zipf.open(fname, 'w', force_zip64=force_zip64) as fil:
format.write_array(fil, val)
else:
# First write to a tempfile, then store tempfile in the zipf
def write_text(val, fname, zipf):
fil = utils.TempFile(prefix='pyformex-', suffix='.tmp')
fil.write(val)
fil.flush()
zipf.write(fil.path, arcname=fname)
fil.close()
def write_array(val, fname, zipf):
fil = utils.TempFile(prefix='pyformex-', suffix='.tmp')
format.write_array(fil, val)
fil.flush()
zipf.write(fil.path, arcname=fname)
fil.close()
for key, val in namedict.items():
if isinstance(val, str):
fname = key + '.txt'
if not val.endswith('\n'):
val += '\n'
val = val.encode('latin1')
write_text(val, fname, zipf)
else:
fname = key + '.npy'
val = np.asanyarray(val)
write_array(val, fname, zipf)
[docs]def savePZF(filename, **kargs):
"""Save pyFormex objects to a file in PGF format.
Parameters
----------
filename: :term:`path_like`
Name of the file on which to save the objects. It is recommended
to use an extension '.pzf'.
kargs: keyword arguments
The objects to be saved. Each object will be saved with a name
equal to the keyword argument. The keyword should not end with
an underscore '_', nor contain a double underscore '__'. Keywords
starting with a single underscore are reservecfor special use
and should not be used for any other object.
Notes
-----
Reserved keywords:
- '_camera': stores the current camerasettings
- '_canvas': stores the full canvas layout and camera settings
::
savePZF(filename, _camera=True, _canvas=True, ...)
See also example SaveLoad.
"""
filename = Path(filename)
pf.verbose(1, f"Write PZF file {filename.absolute()}")
save_dict = {}
for k in kargs:
if k.endswith('_') or '__' in k or k=='':
raise ValueError(f"Invalid keyword argument '{k}' for savePZF")
if k.startswith('_'):
if k == '_camera':
kargs[k] = Config(pf.canvas.camera.settings)
elif k == '_canvas':
kargs[k] = pf.GUI.viewports.config()
o = kargs[k]
if hasattr(o, 'pzf_dict'):
clas = o.__class__.__name__
d = utils.prefixDict(o.pzf_dict(), '%s__%s__' % (k, clas))
save_dict.update(d)
else:
pf.verbose(2, f"Object {k} of type {type(o)} can not (yet) "
f"be written to PZF file: skipping.")
# Do not store camera if we store canvas
if '_canvas' in kargs and '_camera' in kargs:
del kargs['_camera']
pf.verbose(2, f"Saving {len(save_dict)} objects to PZF file")
pf.verbose(2, f"Contents: {sorted(save_dict.keys())}")
save_dict['__FORMAT'] = f"PZF {_pzf_version} (created by {pf.fullVersion()})\n"
with zipfile.ZipFile(filename, 'w') as zip:
save2zip(zip, save_dict)
def split_PZF_keys(d):
D = {}
for k in d:
s = k.split('__')
if s[-1].endswith('.txt'):
s[-1] = s[-1][:-4]
if len(s) >= 3:
name, clas, attr = s[:3]
if name not in D:
D[name] = {'class': clas}
od = D[name]
if attr in ('closed',):
# existence means True
od[attr] = True
elif attr in ('eltype',):
# value is string encoded in name
od[attr] = s[3]
elif attr in ('degree',):
# value is int encoded in name
od[attr] = int(s[3])
elif attr=='field':
if 'fields' not in od:
od['fields'] = []
od['fields'].append(k)
else:
if isinstance(d[k], np.ndarray) and d[k].size == 0:
d[k] = None
od[attr] = d[k]
if isinstance(od[attr], np.ndarray) and od[attr].dtype.kind == 'S':
# Convert bytes to unicode
od[attr] = od[attr].astype('U')
#print(type(od[attr]))
for k in D:
if not 'class' in D[k]:
raise ValueError("Invalid PZF file: no 'class' attribute for object'%s'" % k)
return D
[docs]def loadPolyLine(**kargs):
"""Create PolyLine from PZF file dict"""
#print(kargs)
closed = 'closed' in kargs
kargs = utils.selectDict(kargs, ['coords'])
PL = PolyLine(closed=closed, **kargs)
return PL
[docs]def loadBezierSpline(**kargs):
"""Create BezierSpline from PZF file dict"""
closed = 'closed' in kargs
kargs = utils.selectDict(kargs, ['coords'])
PL = PolyLine(closed=closed, **kargs)
return PL
[docs]def loadFields(obj, pzfd, fields):
"""Load fields from PZF file on an object"""
for f in fields:
fldtype, fldname = f.split('__')[3:]
data = pzfd[f]
obj.addField(fldtype, data, fldname)
[docs]def loadAttrib(obj, attrib):
"""Load attrib dict on the object"""
attrib = bytes(attrib)
d = json.loads(attrib)
pf.debug("attribs for %s object: %s" % (obj.__class__.__name__, d), pf.DEBUG.PGF)
obj.attrib(**d)
[docs]def loadPZF(filename):
"""Load pyFormex objects from a file in PGF format.
Parameters
----------
filename: :term:`path_like`
Name of the file from which to load the objects.
It is normally a file with extension '.pzf'.
Returns
-------
dict
A dictionary with the objects read from the file. The keys in the dict
are the object names used when creating the file.
Notes
-----
If the returned dict contains a camera setting, the camera can be
restored as follows::
if '_camera' in d:
pf.canvas.camera.apply(d['_camera'])
pf.canvas.update()
This will however also restore the camera aspect ratio (width/height).
If the canvas where you load has another aspect ratio, the image will
become distorted: a square will look like a rectangle. This may be
a desired feature (like to show a very long, thin object), but in most
cases it is not. Thus you may want to set the camera aspect to the same
value as your canvas, e.g. before applying the loaded settings::
d = loadPZF(filename)
settings = g.get('_camera',None)
if settings:
settings['aspect'] = pf.canvas.aspect
pf.canvas.camera.apply(settings)
pf.canvas.update()
Now your image will come out with the same aspect as when saving.
If the canvas where you load has a larger aspect ratio, you will get
extra space at the sides; if it is much smaller, you may miss some
parts at the sides. Resize the canvas to view everything.
See also example SaveLoad.
"""
filename = Path(filename)
pf.verbose(1, f"Read PZF file {filename.absolute()}")
D = np.load(filename)
d = split_PZF_keys(D)
for k in d:
O = None
dk = d[k]
clas = dk.pop('class')
pf.verbose(2, f"Loading {clas} '{k}'")
fields = dk.pop('fields', None)
attrib = dk.pop('attrib', None)
if clas == 'Config':
if isinstance(dk['settings'], bytes):
dk['settings'] = dk['settings'].decode('latin1')
O = Config()
O.read(dk['settings'])
else:
pf.debug(("OK, I've got the clas"), pf.DEBUG.PGF)
Clas = globals().get(clas, None)
if Clas is None:
utils.warn("Objects of class '%s' can currently not yet be loaded" % clas)
args = [dk.pop(arg) for arg in getattr(Clas, 'pzf_args', [])]
pf.debug("Got %s args, kargs: %s" % (len(args), dk.keys()), pf.DEBUG.PGF)
if clas in []:
print(dk)
O = Clas(*args, **dk)
if O is not None:
if fields:
loadFields(O, D, fields)
if attrib:
loadAttrib(O, attrib)
d[k] = O
return d
def listPZF(filename):
with zipfile.ZipFile(filename, 'r') as fil:
files = sorted(fil.namelist())
return files
def listPZFobj(filename):
files = listPZF(filename)
special = [f for f in files if f.startswith('_')]
names = []
for k in files:
if k.startswith('_'):
continue
s = k.split('__')
if len(s) >= 2:
obj = "%s (%s)" % tuple(s[:2])
if len(names) == 0 or obj != names[-1]:
names.append(obj)
else:
names.append(k)
return names
# End