tFurther subdivided docs into separate files - sphere - GPU-based 3D discrete e… | |
git clone git://src.adamsgaard.dk/sphere | |
Log | |
Files | |
Refs | |
LICENSE | |
--- | |
commit 04aac0dadf0fa7f95aeacb3b80c305a04f8115a4 | |
parent 82467ca3eedc077e028a9ac7ed3e889c8ff93659 | |
Author: Anders Damsgaard <[email protected]> | |
Date: Wed, 5 Dec 2012 13:13:21 +0100 | |
Further subdivided docs into separate files | |
Diffstat: | |
A doc/Makefile | 2 ++ | |
D doc/sphinx/cpp.rst | 6 ------ | |
M doc/sphinx/index.rst | 312 +----------------------------… | |
A doc/sphinx/sphere_internals.rst | 206 +++++++++++++++++++++++++++++… | |
A doc/sphinx/sphere_ref.rst | 6 ++++++ | |
5 files changed, 216 insertions(+), 316 deletions(-) | |
--- | |
diff --git a/doc/Makefile b/doc/Makefile | |
t@@ -0,0 +1,2 @@ | |
+doc: | |
+ $(MAKE) -C sphinx/ | |
diff --git a/doc/sphinx/cpp.rst b/doc/sphinx/cpp.rst | |
t@@ -1,6 +0,0 @@ | |
-C++ reference | |
-============= | |
-.. doxygenclass:: DEM | |
- :members: | |
- | |
- | |
diff --git a/doc/sphinx/index.rst b/doc/sphinx/index.rst | |
t@@ -20,316 +20,8 @@ Contents: | |
introduction | |
dem | |
python_api | |
- cpp | |
- | |
- | |
- | |
-sphere work flow | |
-================ | |
-After compiling the \texttt{SPHERE} binary (see sub-section \ref{subsec:compil… | |
- \item Setup of particle assemblage, physical properties and conditions… | |
- \item Execution of \texttt{SPHERE} software, which simulates the parti… | |
- \item Inspection, analysis, interpretation and visualization of \textt… | |
- | |
-\subsection{The \texttt{SPHERE} algorithm} | |
-\label{subsec:spherealgo} | |
-The \texttt{SPHERE}-binary is launched from the system terminal by passing the… | |
-#. System check, including search for NVIDIA CUDA compatible devices (\texttt{… | |
- | |
-#. Initial data import from binary input file (\texttt{main.cpp}). | |
- | |
-#. Allocation of memory for all host variables (particles, grid, walls, etc.) … | |
- | |
-#. Continued import from binary input file (\texttt{main.cpp}). | |
- | |
-#. Control handed to GPU-specific function \texttt{gpuMain(\ldots)} (\texttt{d… | |
- | |
-#. Memory allocation of device memory (\texttt{device.cu}). | |
- | |
-#. Transfer of data from host to device variables (\texttt{device.cu}). | |
- | |
-#. Initialization of Thrust\footnote{\url{https://code.google.com/p/thrust/}} … | |
- | |
-#. Calculation of GPU workload configuration (thread and block layout) (\textt… | |
- | |
-#. Status and data written to \verb"<simulation_ID>.status.dat" and \verb"<sim… | |
- | |
-#. Main loop (while \texttt{time.current <= time.total}) (functions called in … | |
- | |
- | |
- #. \label{loopstart}CUDA thread synchronization point. | |
- | |
- #. \texttt{calcParticleCellID<<<,>>>(\ldots)}: Particle-grid hash value calc… | |
- | |
- #. CUDA thread synchronization point. | |
- | |
- #. \texttt{thrust::sort\_by\_key(\ldots)}: Thrust radix sort of particle-gri… | |
- | |
- #. \texttt{cudaMemset(\ldots)}: Writing zero value (\texttt{0xffffffff}) to … | |
- | |
- #. \texttt{reorderArrays<<<,>>>(\ldots)}: Reordering of particle arrays, bas… | |
- | |
- #. CUDA thread synchronization point. | |
- | |
- #. Optional: \texttt{topology<<<,>>>(\ldots)}: If particle contact history i… | |
- | |
- #. CUDA thread synchronization point. | |
- | |
- #. \texttt{interact<<<,>>>(\ldots)}: For each particle: Search of contacts i… | |
- | |
- #. CUDA thread synchronization point. | |
- | |
- #. \texttt{integrate<<<,>>>(\ldots)}: Updating of spatial degrees of freedom… | |
- | |
- #. CUDA thread synchronization point. | |
- | |
- #. \texttt{summation<<<,>>>(\ldots)}: Particle contributions to the net forc… | |
- | |
- #. CUDA thread synchronization point. | |
- | |
- #. \texttt{integrateWalls<<<,>>>(\ldots)}: Updating of spatial degrees of fr… | |
- | |
- #. Update of timers and loop-related counters (e.g. \texttt{time.current}), … | |
- | |
- #. If file output interval is reached: | |
- | |
- \item Optional write of data to output binary (\verb"<simulation_ID>.o… | |
- \item Update of \verb"<simulation_ID>.status#..bin" (\texttt{device.cu… | |
- | |
- \item Return to point \ref{loopstart}, unless \texttt{time.current >= ti… | |
- | |
- | |
-#. \label{loopend}Liberation of device memory (\texttt{device.cu}). | |
- | |
-#. Control returned to \texttt{main(\ldots)}, liberation of host memory (\text… | |
- | |
-#. End of program, return status equal to zero (0) if no problems where encoun… | |
- | |
- | |
- | |
-Sphere algorithm | |
-================ | |
-The \texttt{SPHERE}-binary is launched from the system terminal by passing the… | |
- | |
-#. System check, including search for NVIDIA CUDA compatible devices (\texttt{… | |
- | |
-#. Initial data import from binary input file (\texttt{main.cpp}). | |
- | |
-#. Allocation of memory for all host variables (particles, grid, walls, etc.) … | |
- | |
-#. Continued import from binary input file (\texttt{main.cpp}). | |
- | |
-#. Control handed to GPU-specific function \texttt{gpuMain(\ldots)} (\texttt{d… | |
- | |
-#. Memory allocation of device memory (\texttt{device.cu}). | |
- | |
-#. Transfer of data from host to device variables (\texttt{device.cu}). | |
- | |
-#. Initialization of Thrust\footnote{\url{https://code.google.com/p/thrust/}} … | |
- | |
-#. Calculation of GPU workload configuration (thread and block layout) (\textt… | |
- | |
-#. Status and data written to \verb"<simulation_ID>.status.dat" and \verb"<sim… | |
- | |
-#. Main loop (while \texttt{time.current <= time.total}) (functions called in … | |
- | |
- | |
- #. \label{loopstart}CUDA thread synchronization point. | |
- | |
- #. \texttt{calcParticleCellID<<<,>>>(\ldots)}: Particle-grid hash value calc… | |
- | |
- #. CUDA thread synchronization point. | |
- | |
- #. \texttt{thrust::sort\_by\_key(\ldots)}: Thrust radix sort of particle-gri… | |
- | |
- #. \texttt{cudaMemset(\ldots)}: Writing zero value (\texttt{0xffffffff}) to … | |
- | |
- #. \texttt{reorderArrays<<<,>>>(\ldots)}: Reordering of particle arrays, bas… | |
- | |
- #. CUDA thread synchronization point. | |
- | |
- #. Optional: \texttt{topology<<<,>>>(\ldots)}: If particle contact history i… | |
- | |
- #. CUDA thread synchronization point. | |
- | |
- #. \texttt{interact<<<,>>>(\ldots)}: For each particle: Search of contacts i… | |
- | |
- #. CUDA thread synchronization point. | |
- | |
- #. \texttt{integrate<<<,>>>(\ldots)}: Updating of spatial degrees of freedom… | |
- | |
- #. CUDA thread synchronization point. | |
- | |
- #. \texttt{summation<<<,>>>(\ldots)}: Particle contributions to the net forc… | |
- | |
- #. CUDA thread synchronization point. | |
- | |
- #. \texttt{integrateWalls<<<,>>>(\ldots)}: Updating of spatial degrees of fr… | |
- | |
- #. Update of timers and loop-related counters (e.g. \texttt{time.current}), … | |
- | |
- #. If file output interval is reached: | |
- | |
- * Optional write of data to output binary (\verb"<simulation_ID>.outpu… | |
- * Update of \verb"<simulation_ID>.status#..bin" (\texttt{device.cu}). | |
- | |
- #. Return to point \ref{loopstart}, unless \texttt{time.current >= time.tota… | |
- | |
- | |
-#. \label{loopend}Liberation of device memory (\texttt{device.cu}). | |
- | |
-#. Control returned to \texttt{main(\ldots)}, liberation of host memory (\text… | |
- | |
-#. End of program, return status equal to zero (0) if no problems where encoun… | |
- | |
- | |
- | |
-The length of the computational time steps (\texttt{time.dt}) is calculated vi… | |
-\begin{equation} | |
-\label{eq:dt} | |
-\Delta t = 0.17 \min \left( m/\max(k_n,k_t) \right) | |
-\end{equation} | |
-where $m$ is the particle mass, and $k$ are the elastic stiffnesses. This equa… | |
- | |
-\subsubsection{Host and device memory types} | |
-\label{subsubsec:memorytypes} | |
-A full, listed description of the \texttt{SPHERE} source code variables can be… | |
- | |
-The floating point precision operating internally in \texttt{SPHERE} is define… | |
- | |
-Three-dimensional variables (e.g. spatial vectors in $E^3$) are in global memo… | |
- | |
-\begin{figure}[htbp] | |
-\label{fig:memory} | |
-\begin{center} | |
-\begin{small} | |
-\begin{tikzpicture}[scale=1, node distance = 2cm, auto] | |
- % Place nodes | |
- \node [harddrive] (freadbin) {\textbf{Hard Drive}:\\[1mm] Input binary: \v… | |
- | |
- \node [processor, below of=freadbin, node distance=2cm] (cpu) … | |
- | |
- \node [harddrive, below of=cpu, node distance=2.5cm] (fwritebin) {\textbf{… | |
- | |
- \node [mem, right of=cpu, node distance=2.5cm] (host) {\textbf… | |
- | |
- \node [mem, right of=host, node distance=3.5cm] (textures) {\t… | |
- \node [mem, above of=textures, node distance=2cm] (device) {\t… | |
- \node [mem, below of=textures, node distance=2cm] (constant) {… | |
- | |
- | |
- \node [processor, right of=textures, node distance=4cm] (gpu) … | |
- | |
- | |
- \draw node [above of=gpu, shape aspect=1, diamond, draw, node distance… | |
- \draw node [below of=gpu, shape aspect=1, diamond, draw, node distance… | |
- | |
- \node [mem, above of=gpu, node distance=3cm] (local) {\textbf{Local re… | |
- | |
- \node [mem, below of=gpu, node distance=3cm] (shared) {\textbf{Sha… | |
- | |
- % Place hardware description | |
- \node [above of=freadbin, node distance=1.5cm] {\large Host system}; | |
- %\node [above of=device, node distance=4cm] {\Large CUDA device}; | |
- \node [at={(7.0,1.5)}] {\large CUDA device}; | |
- | |
- \node [at={(4.0,0.8)}, rotate=90] {PCIe $\times$16 Gen2}; | |
- \path [draw] (4.3, 2.0) -- (4.3,-0.5); | |
- \path [draw] (4.3,-3.5) -- (4.3,-6.5); | |
- | |
- \node [at={(6.0,-6.3)}] {Off-chip}; | |
- | |
- \path [draw, gray] (8.0, 0.5) -- (8.0,-0.5); | |
- \path [draw, gray] (8.0,-4.5) -- (8.0,-6.5); | |
- | |
- \node [at={(10.0,-6.3)}] {On-chip}; | |
- | |
- % Draw lines | |
- \path [draw, -latex', thick] (freadbin) -- (cpu); | |
- | |
- \path [draw, -latex'] (cpu) -- (host); | |
- \path [draw, -latex'] (host) -- (cpu); | |
- | |
- \path [draw, -latex', dashed] (host) -- (device); | |
- \path [draw, -latex', dashed] (device) -- (host); | |
- | |
- \path [draw, -latex', dashed] (host) -- (constant); | |
- \path [draw, -latex', dashed] (constant) -- (host); | |
- %\path [draw, -latex'] (constant) -- (device); | |
- | |
- \path [draw, -latex', dashed] (host) -- (textures); | |
- \path [draw, -latex', dashed] (textures) -- (host); | |
- | |
- %\path [draw, -latex', dashed] (host) -- (shared); | |
- %\path [draw, -latex', dashed] (shared) -- (host); | |
- | |
- \path [draw, -latex'] (device) -- (gpu); | |
- \path [draw, -latex'] (gpu) -- (device); | |
- | |
- \path [draw, -latex'] (textures) -- (gpu); | |
- \path [draw, -latex'] (gpu) -- (textures); | |
- \node [at={(7.7,-1.8)}] {\footnotesize Cached}; | |
- \node [at={(7.7,-2.2)}] {\footnotesize reads}; | |
- | |
- \path [draw, -latex'] (constant) -- (gpu); | |
- %\path [draw, -latex'] (gpu) -- (constant); | |
- \node [at={(7.7,-2.9)}, rotate=25] {\footnotesize Cached}; | |
- \node [at={(7.9,-3.2)}, rotate=25] {\footnotesize reads}; | |
- | |
- \path [draw, -latex'] (shared) -- (gpu); | |
- \path [draw, -latex'] (gpu) -- (shared); | |
- \node [at={(9.85,-3.9)}] {\footnotesize Cached reads}; | |
- %\node [at={(8.0,-4.2)}, rotate=45] {\footnotesize reads}; | |
- | |
- \path [draw, -latex'] (local) -- (gpu); | |
- \path [draw, -latex'] (gpu) -- (local); | |
- | |
- | |
- %\path [draw, -latex'] (device) -- (shared); | |
- | |
- \path [draw, -latex', thick] (cpu) -- (fwritebin); | |
- | |
- % Bandwith text | |
- \node [at={(4.2,-2.3)}] (host-dev) {8 GB/s}; | |
- %\node [at={(3,-3)}] (PCIe) {(PCIe Gen2)}; | |
- %\node [at={(6,-2.3)}] (dev-dev) {89.6 GB/s}; | |
- | |
-\end{tikzpicture} | |
-\end{small} | |
- | |
-\caption{Flow chart of system memory types and communication paths. RAM: Rando… | |
-\end{center} | |
- | |
-\end{figure} | |
- | |
- | |
-\paragraph{Host memory} is the main random-access computer memory (RAM), i.e. … | |
- | |
- | |
-\paragraph{Device memory} is the main, global device memory. It resides off-ch… | |
- | |
-\marginpar{Todo: Expand section on device memory types} | |
- | |
-\paragraph{Constant memory} values cannot be changed after they are set, and a… | |
- | |
- | |
- | |
-%\subsection{The main loop} | |
-%\label{subsec:mainloop} | |
-%The \texttt{SPHERE} software calculates particle movement and rotation based … | |
- | |
- | |
-\subsection{Performance} | |
-\marginpar{Todo: insert graph of performance vs. np and performance vs. $\Delt… | |
-\subsubsection{Particles and computational time} | |
- | |
-\subsection{Compilation} | |
-\label{subsec:compilation} | |
-An important note is that the \texttt{C} examples of the NVIDIA CUDA SDK shoul… | |
- | |
-\texttt{SPHERE} is supplied with several Makefiles, which automate the compila… | |
- | |
- | |
+ sphere_ref | |
+ sphere_internals | |
diff --git a/doc/sphinx/sphere_internals.rst b/doc/sphinx/sphere_internals.rst | |
t@@ -0,0 +1,206 @@ | |
+sphere internals | |
+================ | |
+After compiling the *sphere* binary (see sub-section \ref{subsec:compilation})… | |
+ \item Setup of particle assemblage, physical properties and conditions… | |
+ \item Execution of *sphere* software, which simulates the particle beh… | |
+ \item Inspection, analysis, interpretation and visualization of *spher… | |
+ | |
+\subsection{The *sphere* algorithm} | |
+\label{subsec:spherealgo} | |
+The *sphere*-binary is launched from the system terminal by passing the simula… | |
+#. System check, including search for NVIDIA CUDA compatible devices (\texttt{… | |
+ | |
+#. Initial data import from binary input file (\texttt{main.cpp}). | |
+ | |
+#. Allocation of memory for all host variables (particles, grid, walls, etc.) … | |
+ | |
+#. Continued import from binary input file (\texttt{main.cpp}). | |
+ | |
+#. Control handed to GPU-specific function \texttt{gpuMain(\ldots)} (\texttt{d… | |
+ | |
+#. Memory allocation of device memory (\texttt{device.cu}). | |
+ | |
+#. Transfer of data from host to device variables (\texttt{device.cu}). | |
+ | |
+#. Initialization of Thrust\footnote{\url{https://code.google.com/p/thrust/}} … | |
+ | |
+#. Calculation of GPU workload configuration (thread and block layout) (\textt… | |
+ | |
+#. Status and data written to \verb"<simulation_ID>.status.dat" and \verb"<sim… | |
+ | |
+#. Main loop (while \texttt{time.current <= time.total}) (functions called in … | |
+ | |
+ | |
+ #. \label{loopstart}CUDA thread synchronization point. | |
+ | |
+ #. \texttt{calcParticleCellID<<<,>>>(\ldots)}: Particle-grid hash value calc… | |
+ | |
+ #. CUDA thread synchronization point. | |
+ | |
+ #. \texttt{thrust::sort\_by\_key(\ldots)}: Thrust radix sort of particle-gri… | |
+ | |
+ #. \texttt{cudaMemset(\ldots)}: Writing zero value (\texttt{0xffffffff}) to … | |
+ | |
+ #. \texttt{reorderArrays<<<,>>>(\ldots)}: Reordering of particle arrays, bas… | |
+ | |
+ #. CUDA thread synchronization point. | |
+ | |
+ #. Optional: \texttt{topology<<<,>>>(\ldots)}: If particle contact history i… | |
+ | |
+ #. CUDA thread synchronization point. | |
+ | |
+ #. \texttt{interact<<<,>>>(\ldots)}: For each particle: Search of contacts i… | |
+ | |
+ #. CUDA thread synchronization point. | |
+ | |
+ #. \texttt{integrate<<<,>>>(\ldots)}: Updating of spatial degrees of freedom… | |
+ | |
+ #. CUDA thread synchronization point. | |
+ | |
+ #. \texttt{summation<<<,>>>(\ldots)}: Particle contributions to the net forc… | |
+ | |
+ #. CUDA thread synchronization point. | |
+ | |
+ #. \texttt{integrateWalls<<<,>>>(\ldots)}: Updating of spatial degrees of fr… | |
+ | |
+ #. Update of timers and loop-related counters (e.g. \texttt{time.current}), … | |
+ | |
+ #. If file output interval is reached: | |
+ | |
+ \item Optional write of data to output binary (\verb"<simulation_ID>.o… | |
+ \item Update of \verb"<simulation_ID>.status#..bin" (\texttt{device.cu… | |
+ | |
+ \item Return to point \ref{loopstart}, unless \texttt{time.current >= ti… | |
+ | |
+ | |
+#. \label{loopend}Liberation of device memory (\texttt{device.cu}). | |
+ | |
+#. Control returned to \texttt{main(\ldots)}, liberation of host memory (\text… | |
+ | |
+#. End of program, return status equal to zero (0) if no problems where encoun… | |
+ | |
+ | |
+Sphere algorithm | |
+================ | |
+The *sphere*-binary is launched from the system terminal by passing the simula… | |
+ | |
+#. System check, including search for NVIDIA CUDA compatible devices (\texttt{… | |
+ | |
+#. Initial data import from binary input file (\texttt{main.cpp}). | |
+ | |
+#. Allocation of memory for all host variables (particles, grid, walls, etc.) … | |
+ | |
+#. Continued import from binary input file (\texttt{main.cpp}). | |
+ | |
+#. Control handed to GPU-specific function \texttt{gpuMain(\ldots)} (\texttt{d… | |
+ | |
+#. Memory allocation of device memory (\texttt{device.cu}). | |
+ | |
+#. Transfer of data from host to device variables (\texttt{device.cu}). | |
+ | |
+#. Initialization of Thrust\footnote{\url{https://code.google.com/p/thrust/}} … | |
+ | |
+#. Calculation of GPU workload configuration (thread and block layout) (\textt… | |
+ | |
+#. Status and data written to \verb"<simulation_ID>.status.dat" and \verb"<sim… | |
+ | |
+#. Main loop (while \texttt{time.current <= time.total}) (functions called in … | |
+ | |
+ | |
+ #. \label{loopstart}CUDA thread synchronization point. | |
+ | |
+ #. \texttt{calcParticleCellID<<<,>>>(\ldots)}: Particle-grid hash value calc… | |
+ | |
+ #. CUDA thread synchronization point. | |
+ | |
+ #. \texttt{thrust::sort\_by\_key(\ldots)}: Thrust radix sort of particle-gri… | |
+ | |
+ #. \texttt{cudaMemset(\ldots)}: Writing zero value (\texttt{0xffffffff}) to … | |
+ | |
+ #. \texttt{reorderArrays<<<,>>>(\ldots)}: Reordering of particle arrays, bas… | |
+ | |
+ #. CUDA thread synchronization point. | |
+ | |
+ #. Optional: \texttt{topology<<<,>>>(\ldots)}: If particle contact history i… | |
+ | |
+ #. CUDA thread synchronization point. | |
+ | |
+ #. \texttt{interact<<<,>>>(\ldots)}: For each particle: Search of contacts i… | |
+ | |
+ #. CUDA thread synchronization point. | |
+ | |
+ #. \texttt{integrate<<<,>>>(\ldots)}: Updating of spatial degrees of freedom… | |
+ | |
+ #. CUDA thread synchronization point. | |
+ | |
+ #. \texttt{summation<<<,>>>(\ldots)}: Particle contributions to the net forc… | |
+ | |
+ #. CUDA thread synchronization point. | |
+ | |
+ #. \texttt{integrateWalls<<<,>>>(\ldots)}: Updating of spatial degrees of fr… | |
+ | |
+ #. Update of timers and loop-related counters (e.g. \texttt{time.current}), … | |
+ | |
+ #. If file output interval is reached: | |
+ | |
+ * Optional write of data to output binary (\verb"<simulation_ID>.outpu… | |
+ * Update of \verb"<simulation_ID>.status#..bin" (\texttt{device.cu}). | |
+ | |
+ #. Return to point \ref{loopstart}, unless \texttt{time.current >= time.tota… | |
+ | |
+ | |
+#. \label{loopend}Liberation of device memory (\texttt{device.cu}). | |
+ | |
+#. Control returned to \texttt{main(\ldots)}, liberation of host memory (\text… | |
+ | |
+#. End of program, return status equal to zero (0) if no problems where encoun… | |
+ | |
+ | |
+ | |
+The length of the computational time steps (\texttt{time.dt}) is calculated vi… | |
+ | |
+.. math:: | |
+ \Delta t = 0.075 \min \left( m/\max(k_n,k_t) \right) | |
+ | |
+where :math:`m` is the particle mass, and :math:`k` are the elastic stiffnesse… | |
+The time step is set by this relationship in :py:func:`initTemporal`. | |
+This equation ensures that the elastic wave (traveling at the speed of sound) … | |
+ | |
+\subsubsection{Host and device memory types} | |
+\label{subsubsec:memorytypes} | |
+A full, listed description of the *sphere* source code variables can be found … | |
+ | |
+The floating point precision operating internally in *sphere* is defined in \t… | |
+ | |
+Three-dimensional variables (e.g. spatial vectors in `E^3`) are in global memo… | |
+ | |
+ | |
+\paragraph{Host memory} is the main random-access computer memory (RAM), i.e. … | |
+ | |
+ | |
+\paragraph{Device memory} is the main, global device memory. It resides off-ch… | |
+ | |
+\marginpar{Todo: Expand section on device memory types} | |
+ | |
+\paragraph{Constant memory} values cannot be changed after they are set, and a… | |
+ | |
+ | |
+ | |
+%\subsection{The main loop} | |
+%\label{subsec:mainloop} | |
+%The *sphere* software calculates particle movement and rotation based on the … | |
+ | |
+ | |
+\subsection{Performance} | |
+\marginpar{Todo: insert graph of performance vs. np and performance vs. `\Delt… | |
+\subsubsection{Particles and computational time} | |
+ | |
+\subsection{Compilation} | |
+\label{subsec:compilation} | |
+An important note is that the \texttt{C} examples of the NVIDIA CUDA SDK shoul… | |
+ | |
+*sphere* is supplied with several Makefiles, which automate the compilation pr… | |
+ | |
+ | |
+ | |
+ | |
diff --git a/doc/sphinx/sphere_ref.rst b/doc/sphinx/sphere_ref.rst | |
t@@ -0,0 +1,6 @@ | |
+C++ reference | |
+============= | |
+.. doxygenclass:: DEM | |
+ :members: | |
+ | |
+ |