SN DBS e
SN DBS e
SN DBS e
SN Systems Limited
Version 2.0.49
May 11, 2009
Copyright © Sony Computer Entertainment Inc. / SN Systems Ltd, 2003-2009.
"ProDG" is a registered trademark and the SN logo is a trademark of SN Systems Ltd.
"Microsoft", "Visual Studio", "Win32" and "Windows" are registered trademarks of
Microsoft Corporation. "Vista" is a trademark of Microsoft. Other product and
company names mentioned herein may be the trademarks of their respective
owners.
USER GUIDE TO SN-DBS (DISTRIBUTED BUILD SYSTEM) 3
Contents
Contents
1: Introduction 6
Document version history 6
Overview of SN-DBS 6
Building with VSI or makefiles 7
Role of agents 7
Role of the broker 7
Fallback named broker 8
Automatic detection of ProDG tools 8
The SN-DBS build process 8
Multi-processor and multi-core systems 8
Optimizing the use of SN-DBS 9
Object caching 9
Extending SN-DBS 10
SN-DBS user interface 10
Updating the SN-DBS software 10
Versions of SN-DBS 10
Contents of release 10
System requirements 11
Hardware requirements 11
Software requirements 11
Licensing 11
Compatibility 11
Updates and technical support 11
5: Data builds 48
Introduction 48
Requirements for data builds 48
Simple Data Tools 48
Invoking different tools 49
6: Templates 50
Introduction to templates 50
Template format 50
Tool section 50
Files section 51
Includes section 52
Examples and help 52
7: FAQs 53
Introduction 53
Why do I sometimes see timeouts in my SN-DBS build logs? 53
How does SN-DBS know what tool to use? 53
Can I get SN-DBS to prefer a set of dedicated build PCs over those
of my colleagues? 53
How can I run Seti@home or Folding@home without SN-DBS
thinking the PC is always busy? 54
USER GUIDE TO SN-DBS (DISTRIBUTED BUILD SYSTEM) 5
Contents
I've got a really small, lightweight project. It can build locally faster
than SN-DBS can distribute it. Can I build it locally when
performing a distributed build of the entire solution? 54
8: Index 55
USER GUIDE TO SN-DBS (DISTRIBUTED BUILD SYSTEM) 6
Introduction
1: Introduction
2.0.47 Oct. 2008 Updated 'Command line options for DbsRun'. Numerous
minor amendments.
2.0.25 Nov. 2007 Removed 'Agent Discovery Timeout' and 'Max Job Size' from
General tab tab of Configuration dialog.
2.0.19 Oct. 2007 Updated 'Building on the command line'. Changed "VSI.NET"
to "VSI".
2.0.16 Sep. 2007 User guide updated to reflect fact that SN-DBS is now an
unlicensed product.
Overview of SN-DBS
SN Distributed Build System (SN-DBS) is a software product to improve compile times
for large console development projects.
Under carefully controlled conditions the compilation of C/C++ source files is shared
amongst a series of PCs on a local network (referred to as an SN-DBS network). This
utilizes otherwise unused processing capacity, thereby considerably reducing
compilation times for program builds, as shown in the graph below.
USER GUIDE TO SN-DBS (DISTRIBUTED BUILD SYSTEM) 7
Introduction
The time you save can be used to write more code resulting in increased
productivity.
Please note that only compilation tasks are distributed by SN-DBS. Other build tasks,
e.g. linking, will continue to be carried out on your local PC.
Role of agents
Each PC running on the SN-DBS network contains an instance of the SN-DBS
software, known as an agent.
The agent runs in the background and is transparent to the user, with minimal
impact on the normal performance of your PC.
Agents liaise with each other to initiate and distribute builds within the SN-DBS
network.
Each agent can initiate builds as well as offering its own processing power for use by
other agents in the SN-DBS network.
The SN-DBS agent runs as a Windows service. This means it is started when the
system starts up, and runs in the background whether a user is logged-in or not.
Note: We recommend the use of named broker mode for best results.
A broker will be elected automatically by all the agents within the SN-DBS network
but you can override this election if required by specifying a particular PC to be the
broker. This is known as the named broker mode. All other agents in the SN-DBS
network must then be configured with the address of the named broker. See "Named
USER GUIDE TO SN-DBS (DISTRIBUTED BUILD SYSTEM) 8
Introduction
The broker can also use the Withdraw and Restart options to force a remote agent
to withdraw from the SN-DBS network or to restart. This option is available (only to
the broker) on a menu that is accessed by right-clicking the relevant agent in the
Members view.
Object caching
Object caching is a mechanism whereby the results of a compilation can be stored in
order to improve the speed of subsequent builds. It works by maintaining a copy of
all derived objects (typically .o files) as they are built and calculating when it is
possible to re use those cached copies during subsequent builds, rather than re
generating them.
To guarantee the integrity of a build, a number of factors are included into the
unique key that identifies each derived object in the cache. These include the
command-line arguments, the name and location of the source file and the contents
of the source file and its dependencies. This is advantageous because the net effect
of all the source file's dependencies are factored into the key, so changes to any of
its dependencies will result in the cached object becoming invalid, which will
therefore require the derived object to be rebuilt. However, the disadvantage is that
every dependency must be determined and examined before it is possible to
determine if the derived object can be retrieved from the cache.
The cache can be placed on a local disk, or on a network share. Keeping the cache
on a local disk means that it is as fast as possible for the local user, but no-one else
in the team can take advantage of the cache contents. Keeping it on a network share
means that users working on the same project can take advantage of each others'
recent builds, but each cache access takes place over the network, which is
somewhat slower than local disk access, even on a Gigabit network. For more
information on how to configure use of the cache, see "Cache Control tab" on page
35.
Note: The cache storing algorithm takes into account the current working directory
and file paths. This means that you can only use shared cache writes if you have the
same directory structure.
USER GUIDE TO SN-DBS (DISTRIBUTED BUILD SYSTEM) 10
Introduction
Note: When building a project from a local cache, the overall load on the network,
and on other agents, is greatly reduced. This frees up those resources for other
users, hence providing an improvement for the team as a whole.
Extending SN-DBS
By default SN-DBS only supports tools from SN Systems and SCEI. You can however,
extend the use of SN-DBS so that it distributes the work of third-party command-line
tasks. See "Data builds" on page 48 for more information.
Note: This automatic update facility can only be initiated from the broker PC and will
not work from any other agent.
Versions of SN-DBS
SN-DBS identifies the current version of each agent in the SN-DBS network and
rejects those that are running an incompatible version. If an agent attempts to
communicate with a broker with a more recent version, the broker will push out an
update if one is available. See "Automatically updating the SN-DBS software" on page
45. If there is no update available and the agent is incompatible with the broker, it
will not be able to join the network. You must then upgrade the SN-DBS software to
at least the minimum compatible version required by the broker.
Contents of release
This release contains the following tools:
SN-DBS Network View (user interface) (dbsview.exe)
SN-DBS agent (dbsagent.exe)
SN-DBS build client (dbsbuild.exe)
SN-DBS make wrapper (dbsrun.exe) for use with makefile-based projects
SN-DBS utility tool (dbsutil.exe)
SN-DBS service controller (dbssrvctrl.exe)
USER GUIDE TO SN-DBS (DISTRIBUTED BUILD SYSTEM) 11
Introduction
System requirements
Hardware requirements
Intel Pentium 4 (or equivalent) or higher processor
Network adapter
512 MB of RAM
At least 30 MB of hard disk space for the initial install. The amount of space
used by the file cache will depend on the size of your projects.
Software requirements
Windows XP Pro (32-bit and 64-bit) and Windows Vista Business, Enterprise or
Ultimate (32-bit and 64-bit)
.NET 2.0 runtime
Visual Studio Integration v1.7.5 or higher
ProDG or SCEI build tools
Licensing
For each PC whose agent is either initiating or taking part in distributed builds, a
suitable license will be required for the appropriate build tools.
Compatibility
SN-DBS is compatible with:
ProDG for PlayStation®2 v3.2 containing ps2cc v1.2.4.9 or greater
SNC Compiler for PlayStation®2 containing ps2snc build 793 or greater
ProDG for PSP v1.1 patch 1 or greater
SNC PPU Toolchain for PLAYSTATION®3 v1.0.15 or greater
PLAYSTATION®3 Programmer Tool Toolchain (Windows) v4.0.2 or greater
Installing SN-DBS
Please note that SN-DBS version 2.x can be installed alongside version 1.x. This
allows you to try out version 2.x without having to update all machines at once.
1. Before installing SN-DBS you must have the Microsoft .NET framework version
2.0 installed. This is available from Microsoft or through Windows Update.
2. Install a suitable version of VSI for SN-DBS if you intend to initiate builds from
this PC. See "Software requirements" on page 11 for details.
3. Download and run the installer executable for SN-DBS. The installer will set up
the following environment variables:
SN_COMMON_PATH to point to the folder containing the SN-DBS files, e.g.
SN_COMMON_PATH=C:\Program Files\SN Systems\Common.
PATH will have the %SN_COMMON_PATH%\bin directory added.
4. Click Next.
USER GUIDE TO SN-DBS (DISTRIBUTED BUILD SYSTEM) 14
Installation and getting started
5. Click Install to begin the installation. When the installation is complete the
following dialog will be displayed.
File locations
The main executable and support files for SN-DBS are installed into
%SN_COMMON_PATH%\bin.
The help files are installed into %SN_COMMON_PATH%\help.
The following folder will also be created:
%SN_COMMON_PATH%\SN-DBS
This contains the SN-DBS settings and log files. The tool templates are also installed
here.
USER GUIDE TO SN-DBS (DISTRIBUTED BUILD SYSTEM) 15
Installation and getting started
If your PC has multiple network adapters with valid IP addresses, you must specify
the interface for SN-DBS to use. Select File > Configure from the main menu, select
the Network tab and set the Network Adapter option to the correct network
interface. Consult your system administrator if necessary.
We do not recommend that you change the Port Number because this is the number
that allows all the agents to locate each other and the broker. If you do change it
your agent will be removed from the main SN-DBS network. See “Advanced
configuration” on page 43.
USER GUIDE TO SN-DBS (DISTRIBUTED BUILD SYSTEM) 16
Building with SN-DBS
Once connected, it shows one blue dot in the upper right plus 3 gray dots.
When the user has initiated a build, the blue dot moves around the four corners
clockwise.
When the local agent is building on behalf of someone else, the colors are reversed,
and the single gray dot moves around the four corners anti clockwise.
You can open the SN-DBS user interface and view the status of the SN-DBS network
by double-clicking the SN-DBS icon.
The SN-DBS main window initially shows information on the agents present in your
SN-DBS network, in a Members tab. See "Viewing members information" on page 19.
If you subsequently initiate a build from this PC, three further tabs are shown: Build
Activity, Build Progress and Project Progress. See "Monitoring the build" on page
27.
Configure Enables you to configure SN-DBS options such as making the local
machine a preferred broker. See “SN-DBS configuration and
administration” on page 32.
Update Local Forces a re-scan of the system for the locally installed tools.
Tools
Local Tools Displays the tools that have been detected on your PC. See "Viewing
available build tools" on page 22.
Apply Update Enables you to apply an update for which notification has already
been received.
Stop Service Stops the SN-DBS agent service and then quits the SN-DBS user
interface.
Exiting SN-DBS
If you wish to stop the SN-DBS user interface you must select Exit from the
notification area icon shortcut menu.
Note: Clicking on the title bar close button of an open SN-DBS window will only
cause the program to be minimized back to the notification area. Exiting the user
interface leaves the agent service running and you can still initiate and serve builds.
If it is necessary to stop the agent service as well, you must select Stop Service from
the shortcut menu.
See "SN-DBS shortcut menu" on page 16.
File menu
Configure Enables you to configure SN-DBS options such as making the local
machine a preferred broker. See “SN-DBS configuration and
administration” on page 32.
Update Local Forces the agent to check for local tools and report the findings to
Tools other agents on the SN-DBS network. This is done automatically at
regular intervals but this option allows you to instigate the action
on demand, e.g. when you have just installed a new tool.
Local Tools Displays the tools that have been detected on your PC. See "Viewing
available build tools" on page 22.
Apply Update Enables you to apply an update for which notification has already
been received.
Send Update This menu previously provided a way to push out updates through
the broker but this mechanism has now changed. Please see
"Automatically updating the SN-DBS software" on page 45. The
entry has been left in to provide information about the change.
Send Log File Send log file of agent's activity to SN Systems support.
Close Close SN-DBS main window. Note that this only causes the program
to be minimized back to the notification area. The agent service will
still be running and you can still initiate and serve builds.
View menu
Show User Name If checked (and available), the agent names displayed will be those
of the users currently logged in. This applies to the Members,
Activity and Progress views.
Help menu
The local agent details are displayed in blue. If the member details are grayed out
then the agent is unable to provide build services; this may be because the user has
chosen not to participate in builds or they do not have the required tool installed.
The name of the SN-DBS network broker appears in a panel in the status bar at the
bottom of the window.
Agent When SN-DBS is first installed, displays the host name of the PC running as
an agent. You can change this as required by editing the Name property on
the General tab. See "General tab" on page 32.
Time in Shows how long the agent has been (un)available, busy, building, or
this state working for someone else.
USER GUIDE TO SN-DBS (DISTRIBUTED BUILD SYSTEM) 20
Building with SN-DBS
Clicking the Minimize button minimizes the SN-DBS window to the task bar. Clicking
the Close button closes the SN-DBS window, but leaves the program running in the
notification area.
Using the Exclude from all builds menu option you can exclude that agent from all
of your builds. This can be useful if, for example, that agent's PC has expired ProDG
licenses, or is suffering some other temporary condition that is affecting your
builds. If an agent has been excluded from all builds, it will be shown as Excluded
from build in the Members window:
The menu option Include in all builds allows you to include that agent in all your
builds. This is necessary if you have previously excluded an agent from builds due
to some transient condition, but later want to make use of their PC when the
condition has been resolved.
USER GUIDE TO SN-DBS (DISTRIBUTED BUILD SYSTEM) 21
Building with SN-DBS
Note that this is a local setting for your PC only; it has no effect on the remote
agent, but rather stops your local agent from sending build jobs to that remote
agent. The include/exclude settings are persisted, so will remain in effect until you
apply further changes via the Members tab right-click menu.
Using the normal selection methods you can also select more than one agent to
Exclude from or Include in all builds. You can also use the Exclude for and Include
for options to either exclude selected agents from builds or include agents in
builds, using the specified tools.
The Withdraw and Restart menu options are only available to the broker. They are
used to force remote agents to withdraw from the SN-DBS network or to restart.
The Configure... option allows the Named Broker to configure agents remotely.
Please see "Remote configuration" on page 37 for more details.
3. To view the available tools from another agent, on the SN-DBS main window,
double-click the agent whose tools you wish to view.
USER GUIDE TO SN-DBS (DISTRIBUTED BUILD SYSTEM) 23
Building with SN-DBS
2. Information about agents with the specified tool will be displayed on the SN-DBS
main window. Non-matching agents will be grayed out. If the selected template
is for a 64-bit executable, then all agents running on 32-bit systems will be
grayed out and their status will be set to "64-bit tools not supported".
Alternatively, you can select Build Project using SN-DBS or Build Solution using
SN-DBS from the Visual Studio Build menu. Note there are Build and Rebuild
options for both projects and solutions.
The build ouput will be piped back to the command line shell and <Ctrl+C> will exit
the build as normal.
USER GUIDE TO SN-DBS (DISTRIBUTED BUILD SYSTEM) 24
Building with SN-DBS
Option Action
-aa Always use alternative dependency tracking for problematic files. This is
useful when a macro-based include statement resolves differently when
included in different source files.
-na Disables the alternative fallback mechanisms entirely, relying only on
scanning the source-files to determine dependencies; this mode should
be used for the fastest build time, but will most likely cause builds to
fail if SN-DBS cannot determine dependencies.
-wa Issues warnings to stderr when falling back to either of the alternative
dependency-scanning mechanisms
-wi Issues warnings to stderr when it encounters any #include MACRONAME
constructs.
USER GUIDE TO SN-DBS (DISTRIBUTED BUILD SYSTEM) 25
Building with SN-DBS
Option Action
-aa Always use alternative dependency tracking for problematic files.
This is useful when a macro-based include statement resolves
differently when included in different source files.
-ch Causes cache hits to show the text "DBSBUILD: CACHE HIT:
<filename>" on stdout.
-chx Show cache hits but also prints the job hash.
-di Prints out the discovered dependency tree.
-gt Allows generic tools. See "Simple Data Tools" on page 48 for
details.
--help Prints the usage information.
-k Keep going on errors.
Option Action
-aa Always use alternative dependency tracking for problematic files. This
is useful when a macro-based include statement resolves differently
when included in different source files.
-ch Causes cache hits to show the text "DBSBUILD: CACHE HIT: <filename>"
on stdout.
-chx Shows cache hits but also prints the job hash.
-gt Allows generic tools. See "Simple Data Tools" on page 48 for details.
--help Prints the usage information.
-log Writes log events for this build to the given log file (otherwise uses the
<logfile> standard log file).
-na Disables the alternative fallback mechanisms entirely, relying only on
scanning the source-files to determine dependencies; this mode should
be used for the fastest build time, but will most likely cause builds to
fail if SN-DBS cannot determine dependencies.
-timeout How long the build will be allowed to run before exiting.
<mins>
The project name is optional but useful when looking at the SN-DBS UI. The jobs in
the script do not need to be prefixed with 'dbsbuild' as they do when issuing them
on the command line. You should be able to give the script a .bat extension and run
it locally like a batch file. The instance of dbsbuild will return when it has received a
build result for each job in the script. This is how VSI issues jobs to SN-DBS, using
one script file per project.
For simple makefiles you can create a script with a just a few lines. The following
lines are provided as an example only. The variables CC, CFLAGS, INCLUDES, OBJS,
LD, LDFLAGS, LIBPATHS, and LIBS should be familiar to someone used to working
with makefiles.
BUILD_SCRIPT = MyScript.dbs
DBSBUILD = DbsBuild
all: $(TARGET).elf
$(BUILD_SCRIPT):
@for %%f in ($(patsubst %.o,%,$(OBJS))) do \
echo $(CC) $(CFLAGS) $(INCLUDES) -c %%f.c -o %%f.o
>>$(BUILD_SCRIPT)
$(TARGET).elf: $(BUILD_SCRIPT)
$(DBSBUILD) -p MyProject -s $(BUILD_SCRIPT)
$(LD) $(LIBPATHS) -o $@ $(OBJS) $(LIBS) $(LDFLAGS)
Alternatively, you could call DbsRun inside the makefile by prefixing it on the call to
DbsBuild. Specifying it on the command line though allows you to have multiple
calls to DbsBuild without needing to invoke DbsRun each time. You can also use
make with the -j flag to issue multiple scripts to SN-DBS in parallel. Note that the
script extension of .dbs has no significance other than for identification. SN-DBS
does not have any registered file extensions.
Status bar
Once the SN-DBS network has been discovered, the status bar of the Network View
will show information about the network, namely the number of agents and the
current broker.
When a build is initiated, the status bar will change to show statistics for the build.
These show the time taken, the number of files processed, and the number of
agents used. The latter two values will be 0, whilst the files are being synchronized
on the remote file caches. This action is indicated in the status bar.
Once the caches are synchronized, the details are updated with the number of
agents being used, and the number of files built versus the total number of files.
As each file is compiled the following information is displayed (for that file):
The list will automatically scroll downwards as more results are added. This behavior
will stop if you scroll up to another part of the list. Scrolling back down to the
bottom of the list will cause the list to automatically scroll again.
USER GUIDE TO SN-DBS (DISTRIBUTED BUILD SYSTEM) 29
Building with SN-DBS
The statistics at the bottom indicate the duration of the compiled build, the number
of files that have been processed and the number of agents that were used to
compile the build.
At the end of the build, if you click the Results column header all the warning and
error results and failures will be placed together.
For each agent there are two or more bars. The top (gray) bar reports file
synchronizing. Below this is one bar for each CPU the remote agent is using. In the
above screenshot each agent is using 4 CPUs.
Error handling
If a compilation fails, error messages are returned in the same way as if the
compilation were being run locally. SN-DBS will relay textual output from the
compiler back to the initiating PC, where it is displayed in the VSI output window.
If a PC is switched off during compilation or an agent exits, SN-DBS will reschedule
the job. If this fails it will execute the compilation locally. In these circumstances a
build will always be performed but the overall performance may be impaired.
Build jobs that fail due to compiler license errors will be rescheduled to other
agents. When an agent fails a build due to a license error, it will stop reporting that
compiler as one of its available tools.
USER GUIDE TO SN-DBS (DISTRIBUTED BUILD SYSTEM) 32
SN-DBS configuration and administration
Configuring agents
If you wish to amend information about the agent and/or the way your PC works
within SN-DBS, select Configure from the notification area icon shortcut menu (see
"SN-DBS shortcut menu" on page 16) or the File menu (see "File menu" on page 18).
The Configuration dialog is divided into tabbed pages.
For each tab, amend the details as necessary and click Apply. Where relevant,
amendments will be displayed on the Members tab of the SN-DBS main window. See
”Viewing members information” on page 19.
General tab
Agent Settings
Name Name that appears in the Members tab of the main window.
Label The label for this agent, which is an arbitrary tag for easy
identification. It may help to mark all the machines in a particular
physical location, or to identify all the QuadCores, for example.
Multiple labels can be specified by using a semi-colon to separate
them, e.g. label1;label2;label3.
Provide Build Check this option and this PC will be used to provide build services
USER GUIDE TO SN-DBS (DISTRIBUTED BUILD SYSTEM) 33
SN-DBS configuration and administration
Services and initiate builds. If unchecked it will only be used to initiate builds.
Logging
Max. Log Size Displays the maximum size the log file will grow to on disk.
(Mbytes)
Updates
Allow Silent Determines whether the agent will be notified about automatic
Updates updates or whether they will occur transparently.
DLL Detection
Excluded DLLs A list of regular expressions describing DLLs to be excluded from the
DLL detection for generic tools. See "Simple Data Tools" on page 48
for details.
Network tab
Addressing
Port Number The port number that will be used by the TCP/IP connection. This
will be set to 10003 by default. This value must be the same for all
PCs within the SN-DBS network.
Network Adapter If you have more than one active network adapter, specify the
preferred network interface.
IP Address IP address of this agent. If you have more than one IP address
bound to the network card, specify the preferred address.
Broker Discovery
Manual Discovery Determines how this agent will locate the broker. If unchecked the
USER GUIDE TO SN-DBS (DISTRIBUTED BUILD SYSTEM) 34
SN-DBS configuration and administration
Set this PC to be Determines whether this agent will be the named broker. Note that
the Named Broker this option will only be available when Manual Discovery is
checked.
Broker Settings
Allow older This controls whether or not the broker will allow older agents to
agents to connect join the network. Agents that are too old to be compatible will still
(if compatible) be rejected.
Allow newer This controls whether or not the broker will allow newer agents to
agents to connect join the network. The newer agents will do their own checks to
make sure they are compatible with the current broker.
Push out updates If checked, agents with a different version will be sent the installer
for the current broker version.
Number of server This controls the maximum number of server cores that the broker
cores to allocate will allocate to each client that initiates a build.
to each agent This option will only be available if Set this PC to be the Named
Broker is checked.
Set Update Allows you to change the message that is sent out along with the
Message update.
CPUs tab
For Locally Specify the CPU usage when initiating a build. See "Specifying the
Initiated Builds number of CPUs used" on page 40.
USER GUIDE TO SN-DBS (DISTRIBUTED BUILD SYSTEM) 35
SN-DBS configuration and administration
Use local CPUs When there are insufficient remote agents to complete the build,
your local PC will be used for compilations as well as remote PCs.
For Remotely Specify the CPU usage when your PC is building on behalf of
Initiated Builds another agent.
Rating Enables you to re-calculate your suitability rating within the SN-
DBS network. See "Recalculating an agent's performance rating" on
page 40.
Delete Enables you to delete either the oldest or the newest cache entries.
oldest/newest Note that if you change this option to oldest, the "entries before"
cache entries text in the Reduce Cache Now section, changes to "entries after".
Monitor cache If checked, SN-DBS periodically scans the cache to determine if it has
reached a preset limit, at which point it will be reduced to a specified
limit. See "Controlling the caching mechanism" on page 43.
Reduce cache Immediately reduces the size of the cache. You can either remove
now entries before a given date or reduce the whole cache to a given size.
This tab controls the thresholds that the SN-DBS agent uses to determine when its
host PC is busy, and thus unable to participate in distributed builds. If any of the
thresholds are exceeded then the busy condition will be triggered. Each threshold
can be set individually.
Note: On Windows Vista it may be necessary to set the Hard Disk Usage threshold to
100% because of the background hard disk usage of Windows Vista.
Hardware Detection
CPU Usage The CPU Usage slider determines how much of the available CPU
power can be given over to other tasks (aside from SN-DBS) before
the agent will decide that the host PC is busy. When CPU usage
exceeds this level, the agent will tell the broker it is busy, no further
jobs will be scheduled to this agent and the agent will abort any
jobs currently in progress.
Hard Disk The Hard Disk Usage slider determines how much of the available
Usage total disk I/O capacity of the host PC can be used for other tasks
before the agent will decide that the host PC is busy. When disk I/O
exceeds this level, the agent will tell the broker it is busy, no further
jobs will be scheduled to this agent and the agent will abort any
jobs currently in progress.
Free RAM The Free RAM slider determines the minimum amount of free RAM
that can be available before the SN-DBS agent will decide that the
host PC is busy. When free memory drops below this level, the agent
will tell the broker it is busy, no further jobs will be scheduled to
this agent, and the agent will abort any jobs currently in progress.
Free Hard Disk The Free Hard Disk Space slider determines the minimum amount
Space of free disk space that can be available before the SN-DBS agent will
decide that the host PC is busy. When available disk space drops
below this level, the agent will tell the broker it is busy, no further
jobs will be scheduled to this agent and the agent will abort any
jobs currently in progress.
Defaults
Excluded Processes
(browse button) Click to specify a list of processes that will be ignored when
determining if the host is busy. See "Excluding processes from 'host
busy'" on page 39.
Colors tab
This tab allows you to change the color scheme used by SN-DBS.
Remote configuration
The Named Broker is able to configure certain settings on remote agents. Settings
that only apply locally, such as the those on the Colors tab, are not available for
remote configuration.
1. On the Named Broker, right-click the agent to configure and select Named
Broker Tasks > Configure.... This will display the configuration dialog
populated with the settings for the remote agent.
USER GUIDE TO SN-DBS (DISTRIBUTED BUILD SYSTEM) 38
SN-DBS configuration and administration
2. Change settings as required. The new values will be sent to the remote agent.
Note that the name and IP address of the remote agent are displayed in the
title bar of the configuration form.
You can edit settings for multiple agents at once. Select the agents to configure,
right-click and select Named Broker Tasks > Configure.... The configuration
dialog will be shown, but with all settings initially disabled. To apply a setting,
select the checkbox for that setting and then modify the value.
When settings are applied, only the active settings are sent to the remote
agents. The text in the title bar shows that you are configuring remote agents.
You can double-check which agents are being configured as they will still be
selected in the Members View window.
Different PCs can obviously have a different number of processors. If you select
a dual-core and a quad-core, the settings will allow you to specify that both PCs
should use 4 CPUs when providing build services. When the dual-core machine
USER GUIDE TO SN-DBS (DISTRIBUTED BUILD SYSTEM) 39
SN-DBS configuration and administration
receives this new setting, it will compare it against its own number of physical
CPUs, and reduce the setting to 2.
Your existing rating will be displayed in the Per CPU Rating box.
2. Click the Recalculate button.
The Performance Rating dialog will appear. Ensure your PC is idle, i.e. it is not
building or running or performing any other CPU-intensive tasks, before clicking
the Calculate button. The cursor will change to the hourglass (wait cursor) for a
few seconds whilst the calculation takes place.
3. When the calculation has finished the new performance rating will be displayed:
4. Click Accept to update the Displayed Rating value on the CPUs tab of the
Configuration dialog.
5. Click OK to accept the new rating.
4. The address of the Fallback Broker can optionally be set to the hostname or IP
Address of another agent that will become the broker if the named broker fails.
For all other agents in the SN-DBS network:
1. Select Configure from the "SN-DBS shortcut menu" on page 16 or the "File
menu" on page 18. Select the Network tab. See "Network tab" on page 33.
2. Set Broker Discovery to Manual Discovery (recommended).
3. Set Address of Named Broker to the IP address of the named broker or the
host name.
4. Make sure that Set the PC to be the Named Broker is unchecked.
USER GUIDE TO SN-DBS (DISTRIBUTED BUILD SYSTEM) 43
SN-DBS configuration and administration
Advanced configuration
If required, you can set up alternative SN-DBS networks, where compilations would
only be shared between members of the relevant network. To do this, in the
Network tab, specify a different Port Number for each SN-DBS network, i.e. each
member of a particular network must have the same Port Number.
The size of the cache is not calculated automatically. Click Update to calculate and
display the current size of the cache.
Cache Control
The Cache Control tab controls how the cache works in normal operation. You can
choose whether or not to read from the cache, write results to it and automatically
monitor it. The default settings are to take no action.
The cache can be located on a local disk or shared server. A local disk will result in
faster access times, but a remote server allows you to share cache results with other
users.
USER GUIDE TO SN-DBS (DISTRIBUTED BUILD SYSTEM) 44
SN-DBS configuration and administration
Note: The caching algorithm takes into account the current working directory and
file paths. This means that your users should each have the same project directory
structure to make best use of a shared object cache.
The Monitor cache facility instructs SN-DBS to periodically scan the cache and
determine if it has reached a preset limit. When this occurs the cache is reduced to
the limit specified. Files are deleted using a 'Least/Most Recently Accessed' list,
depending on the oldest/newest setting in the General Settings section of the Cache
Control tab.
Note: Depending on the size of the cache these operations may take a few minutes.
SN-DBS utility
The program dbsutil.exe allows you to obtain SN-DBS network information and
make certain changes from the command line.
Usage:
dbsutil <options>
Option Action
-check- Reports template information about the given tool. This allows
template<tool> you to see specifics about what files are sent over with the tool
as well as other relevant information. This will first look for a
side-by-side template (see "Templates" on page 50 and then
enumerate through the relevant templates in the SN-
DBS\Templates folder. If no matches are found a generic
template is consructed, as would be used when supplying the -
gt flag (see "Command line options for DbsBuild" on page 25
and "Command line options for DbsRun" on page 26). Evaluation
is done using the current set of environment variables. This
allows you to check which files are being sent out in your
current build environment, which may differ to your normal
system environment. You can specify the full path to the tool, or
just the exectable name and it will be found on the PATH
environment variable. e.g.
-check-template ppu-lv2-gcc
-check-template C:\usr\local\cell\host-win32\ppu\bin\ppu-lv2-
gcc.exe
-clear-fcache Clears the file-cache at the given agent if specified, or else at
[<ipaddr>]|<host>] every remote agent.
-cpu Detect and print CPU type.
-cr <on|off> Turn cache reads on/off.
-cw <on|off> Turn cache writes on/off.
-delete-cache Delete all files in the cache (not currently being accessed).
-get-log <agent> Retrieves a log file from an agent, where:
<day> <output> agent = <ipaddr> | <hostname>
day = -today | -yesterday | -date <YYYY-MM-DD>
output = -stdout | -o <filename>
Examples of use are:
-get-log 192.168.0.50 -today -o C:\temp\log.txt
-get-log build-pc2 -date 2009-02-27 -stdout
Note that the following three options require elevation when running on Windows Vista
with UAC turned on.
-restart-agent Restart the SN-DBS agent service.
-start-agent Starts the SN-DBS agent service (if it is not currently running).
-stop-agent Stops the SN-DBS agent service (if it is currently running).
Note: The option '-reduce-cache 0' is not the same as '-delete-cache' as the former
can leave behind zero-byte files.
USER GUIDE TO SN-DBS (DISTRIBUTED BUILD SYSTEM) 48
Data builds
5: Data builds
Introduction
By default SN-DBS only supports tools from SN Systems and SCEI. You can however
extend the use of SN-DBS so that it distributes the work of third-party command-line
tasks that meet certain requirements.
To achieve this you will need to create templates in the same format as those that
ship with SN-DBS. See "Templates" on page 50 for further details.
For very simple tools which consist of just a single executable, a tool template may
not be needed. See Simple Data Tools on page 48 for further details.
The input file marker will be stripped away when run on the remote agent.
These two methods are compatible with each other, allowing you to specify both an
extension in the template and an input file marker. This means new build systems
can be made simpler than previous setups, but the existing ones do not need to be
modified.
Previous versions supported the special comment marker $$C. This allowed you to
specify input and output file markers, whilst not having this information appear on
the command line. e.g.
bmp2jpg car.bmp $$C $$O:car.jpg
would indicate to SN-DBS that car.jpg was an output file, but the actual command
used is:
bmp2jpg car.bmp
This marker is still supported so you do not need to modify existing setups.
USER GUIDE TO SN-DBS (DISTRIBUTED BUILD SYSTEM) 50
Templates
6: Templates
Introduction to templates
Tool templates are used by SN-DBS to identify tools and their requirements. You can
create your own templates, expanding the list of tools SN-DBS can distribute jobs
for. This allows you to distribute tasks such as texture conversion, or sound
processing.
The tool templates for the currently supported tools are stored in the SN-
DBS\Templates folder.
This is located in:
%SN_COMMON_PATH%\SN-DBS\Templates
You can add new templates here and SN-DBS will display them in the UI. After adding
a new template you will probably need to select File >Update Local Tools from the
UI menu.
Alternatively, you can specify a template alongside the tool executable. This is called
a side-by-side template. This tool will not be shown in the UI, but has the advantage
that it can be automatically updated via source control. A side-by-side template must
have the same name as the executable and be appended with ".sn-dbs-tool.ini". e.g.
for tool.exe the template would be called tool.exe.sn-dbs-tool.ini.
For details about creating templates please see "Template format" on page 50.
Template format
The templates are used to describe a tool so that SN-DBS knows about the necessary
files. Templates also show up in the UI, allowing you to perform actions such as
excluding agents for a given tool. The template is a simple .ini file with 3 sections.
Tool section
This describes some basics about the tool. In most cases, just the search_path and
extension keys should be required. e.g.
[tool]
search_path=%mytools%\bin;C:\utils\bin
extensions=.abc
agent. This is to catch hangs or other related problems. The default timeout is
appropriate to compilation tasks, which may not be long enough for some custom
tools. This setting allows you to override the default 5 minute timeout to something
more appropriate.
Name: search_path
Value: The search path on which to look for the tool executable.
Required: Yes
Purpose: A semi-colon delimited list of folders. Environment variables can be used
between % signs. e.g. C:\psp\bin;%SN_PSP_PATH%\bin
Name: version
Value: The version number of the tool.
Required: No
Purpose: Allows you to specify the version number that the tool must be. This is
currently used to identify the different versions of the PS3 GCC compiler. Different
versions of this tool have the same main file, but different dependencies.
Name: extensions
Value: The set of extensions used by input files for this tool.
Required: No
Purpose: Allows SN-DBS to automatically identify input files from the command line
without the need for special input file markers (see "Invoking different tools" on
page 49). Multiple file extensions can be specified using a semi-colon as a
separator. e.g. .jpg;.bmp;.gif
Name: 64_bit
Value: Yes/No
Required: No (determined on demand)
Purpose: Allows you to override the automatic detection of whether or not a tool is
32 or 64-bit. This test is performed on the main file, but you may have a supporting
file which can only be used on 64-bit systems. The reason SN-DBS doesn’t test all
the files is to allow for tools which can have 32 or 64-bit support files. For example,
the PS3 PPU SNC compiler driver will run either the 32 or the 64-bit version of the
compiler, based on the host system detected. Checking all files in the template
would mean that SN-DBS would only schedule this tool to 64-bit machines.
Name: use_cache
Value: Yes/No
Required: No (defaults to No)
Purpose: If set to Yes, the build results cache may be used to accelerate builds
using this tool. If set to No, the build results cache is never used for this tool. Note:
Caching must be enabled globally for these settings to have effect. See the "Cache
Control tab" on page 35.
Files section
This section lists the files that are required for this tool. A template must specify a
main file, which is the tool invoked on the command line, and then 0 or more
supporting files. These are typically other executables or DLLs. The main file is
specified with a key called ‘main’, and the other files are specified with keys called
file01, file02, and so on. The files can be specified using absolute paths, or as paths
relative to the main file or a folder on the search path.
e.g.
[files]
main=mytool.exe
file01=toolhelper.exe
file02=..\util\subtask.exe
file03=C:\mydlls\libs.dll
USER GUIDE TO SN-DBS (DISTRIBUTED BUILD SYSTEM) 52
Templates
Includes section
This section is optional, and in most cases will not be needed. It is used to specify
the location of files that must already exist on the remote agent for it to be able to
process a job for this tool. The keys in this section are the names of the files, and
their value is the search path for that file. This example is taken from the template
for ps2cc, which requires a license file in order to work:
[includes]
snl0000.txt=%SN_PATH%;%SN_PATH%\bin
7: FAQs
Introduction
The following FAQs are included in this section:
Why do I sometimes see timeouts in my SN-DBS build logs?
How does SN-DBS know what tool to use?
Can I leave the Windows Firewall running and just add exception rules to allow
SN-DBS to work through the firewall?
Can I get SN-DBS to prefer a set of dedicated build PCs over those of my
colleagues?
How can I run Seti@home or Folding@home without SN-DBS thinking the PC is
always busy?
I've got a really small, lightweight project. It can build locally faster than SN-DBS
can distribute it. Can I build it locally when performing a distributed build of the
entire solution?
8: Index
Files section 52
A Filtering the display of members 23
E M
Error handling 31 Monitoring the build 28
Examples and help 53 Multi-processor and multi-core systems 8
Excluding agents from a build 20
Excluding processes from 'host busy' 40 N
Exiting SN-DBS 17
Extending SN-DBS 10 Named broker configuration 42
Named broker tasks 21
Network tab 34
F
Notification area icon 16
Fallback Named Broker 8
FAQs 54 O
File locations 14
File menu 18 Object caching 9
USER GUIDE TO SN-DBS (DISTRIBUTED BUILD SYSTEM) 56
Index
R U
Recalculating an agent's performance rating Updates and technical support 11
41 Updating the SN-DBS software 10
Reduce cache now 45 User Interface reporting 45
Remote configuration 38
Requirements for data builds 49 V
Role of agents 7
Role of the broker 7 Versions of SN-DBS 10
View menu 18
Viewing available build tools 22
S Viewing build activity 30
Simple Data Tools 49 Viewing build history 31
SN-DBS configuration and administration 33 Viewing build progress 28
SN-DBS menu options 17 Viewing members information 19
SN-DBS shortcut menu 16 Viewing Project Progress 30
SN-DBS user interface 10
SN-DBS Utility 47 W
Software requirements 11
Specifying the number of CPUs used 41 Why do I sometimes see timeouts in my SN-
Specifying the working directory 41 DBS build logs? 54
Status bar 28
System requirements 11
Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.
Alternative Proxies: