Thursday, November 19, 2015

TACTIC Custom Tools and Plugins


INTRODUCTION

Our company builds web based casino games such as slot machines, video conferencing poker, roulettes, and all sorts of casino games. We build these games from inception, design, 2d/3d art assets, programming, to the final web site. There is already an existing base of games that needs to have its interface updated, as well as adding new games with new gameplay and eye candy that can attract players. The size of the projects also varies in size, there’s going to be one flagship game every year and a bunch of mid tier games and lots of smaller games. Besides the games itself, our department also need to provide commercials for these games in the form of computer animations. So there is a large variety of projects to consider. Our company have two 3d teams in house, one using 3dsmax and the other with Maya. 3dsmax users are primarily focused on game assets. While maya users could be tasked with games, animations, commericials.


PROJECT STRUCTURE

I started with Tactic version 4.1.3 and decided not to use the templates. The first thing I did is determing the project structure. By default Tactic seems to want the administrators to set each project as the top level. I decided against that and simply set our department(or just an abstract) as the top level. The main objectes to track are the projects, and with each project, depending on type, assets and/or shots. Projects will have pipelines consisting of marketing, concept, 3d, package, and RD, that tracks the project as it moves from department to department. Assets and Shots are a sub type of concept and 3d deprtment and will have pipelines that track assets through 2d/3d processes. 



FILE MANAGEMENT IN MAYA/MAX/NUKE INTEGRATION


Main UI
I did not particularly like the check-in check-out implemented in TACTIC. I spent too much time trying to get it to work or have any actual use. JAVA was being phased out of Chrome during 2014, and the check-in tool saw action for a couple of months, but had too many problems regarding unicode, UNC path and what not. In the end, check-ins were only used for images/videos, and the JAVA checkin tool is abandoned, and we only used HTML5 image/video check-in or check-ins using the Python API.So, instead of file management using check-ins, I developed a system that can check TACTIC for projects, its associated assets and shots, and its pipelines from Maya/Max/Nuke. The system provides folder naming convention and file naming convention based on TACTIC, and versioning based on files in the folder. As it is built with python and QT, I could run this tool from Maya/Max/Nuke. So once the project and its assets are determined and properly inputed into TACTIC, I can open up any DCC tool and the script, where I can find the project > object > process and either do the initial save, or open an existing file. Each time I save with the tool it increments the version. So there is no over writing with the tool. A little bit of info is also retrieved and displayed in the tool for convenience sake as well. 
These buttons filters projects that are currently in progress, ready, or completed projects. 

This button says ‘Update Cache’. Since I need three lists of projects(in progress, ready, complete), I need to query and search TACTIC three times, but that takes too long. So what I did was to write out three separate lists to a table on tactic, and read from their instead. So instead of a search, it simply retrieves the lists. This cache is updated daily, but can also be updated manually with this button. 

Here is the main interface, choosing projects will show which items are available to work on. The user can choose assets or shots. Choosing items will show pipeline processes, in this case, model and rigging. 

These are supplementatry information, the left pane displays project name, project type, project coordinator, start date, end date. The right pane displays item name, item type, assigned, start date, end date.
The first row is the login name, corresponding button logs out the user and shows a login menu. 
The second row displays the generated folder path, the rule here is {project name}/{assets/shots}/{item_type}/{item_name}/{process}/ and then a unified maya/max/nuke folder structure, where its basically an input folder for textures, or plates for nuke, an output folder for renders, a scenes folder for the work files, and a data folder for cache, misc data, hair, garbage, etc. The corresponding button opens the path in file explorer. 
The last row displays the generated filename, the rule here is, {abbreviation of project name}_{abbreviation of item_type}_{item_name}_{abbreviation of process}_{version}_{author}.{mb, max, or nk}. The corresponding button saves the file. The script knows the application that opened it, so it will load the corresponding API(maya.cmds, Maxplus, or Nuke), and use the appropriate command. It will also set the project environment for each application.
This is the files pane, picking any file and press open, opens the file. Publish could either check the file into TACTIC, or copy the file up a level as a master file. Function not yet determined, and not really essential.
From here on out, a lot of automation can be introduced, for example quick renders directly to TACTIC, render farm outputs to TACTIC. Which brings us, to my next tool.

DAILIES TOOL V01

Instead of checking in files to TACTIC for version control, we adapted the checkin for Quality control/dailies. This tool is a custom layout in TACTIC that first displays a tile view of current projects, which can be quickly sorted and filtered, and then displays the assets and shots of the selected project. The display is two dimensional, where left and right cycles through the items, while up and down cycles through all the images/videos of that particular items through out its pipeline. This tool combines custom layout, python, javascript, jquery, and reveal.js. 


The structure of this one is quite interesting, it is a custom layout that loads a list of project using python through mako. The python script builds the main page by appending HTML to a massive string. Using the Javascript Behaviors in custom layout, it takes project information and filter settings from the page and runs server.execute_command() to execute another python script that builds the reveal.js interface. This python script generates a HTML string and returns it to JS which is prepended into the top of the DOM. 


Figure 1 Dailies Selection Page

The top left tool bar are filters to filter out projects based on project type and item type. Choosing item type will show only items of that type in the dailies page.  


Figure 2

 Title PageClicking on a project will enter the dailies page. Here is the title page. 


Figure 3 Overview

By pressing ESC we enter overview of the dailies page. Since its hard to demonstrate the structure in each slide, we’ll look at the over all structure of the dailies tool. As we can see, items are listed horizontally, while history of each item is listed vertically. History is sorted by time, and then by pipeline process. So the newest image or video of the latest process will always be on top. Navigation are using arrow keys or the joy pad at the right bottom of the screen.  



Figure 4 Image View

This is the actual dailies view based on reveal js. One of the challenges is gettings CSS to work with reveal js AND TACTIC. Formatting CSS under a TACTIC table is quite a challenge. In the end, I just prepend everything to the top of the DOM and attach a separate CSS file while in the dailies view. Loading images and videos takes at least 2 to 3 seconds, as it needs to search through project type > project items > snapshots > files. 

So this is our automated quality control and dailies tool I have implemented. Currently in development is a better lightbox and UI design where I can zoom in/out and pan the images. The standalone tool is built already but I haven’t had time to integrate it into TACTIC yet. 

RENDERFARM UTILIZATION TO TACTIC


Our renderfarm, also setup by me, are blades running ESXI hypervisors managed with VSphere. Although blade and cluster utilization information are readily availabe in VSphere, management required a more user friendly interface then logging into VSphere Center and finding the settings to display utilization data. Since it is only a tool to assess farm capacity, it does not require real time data. Instead, I used PyVmomi to write out Vsphere data to TACTIC daily. From there, I used Highcharts.js to render out Cluster Utilization. The top chart are 16 blades divided into 4 clustes allocated to 4 departments. So each department has access to at least one cluster. The bottom chart are retired workstations serving as additional capacity.



Highcharts is pretty cool in that I can zoom in to a particular range in time.  


MODEL, ANIMATION DATABASE


I have also built an model and animation database into our TACTIC Server. Models and animations are acquired from both free and paid websites.  Using Selenium to scrape files, thumbnails, and information. All of these is then compiled into TACTIC where it can be keyword searched, or browsed by category. The top left bar filters out the models based on model type, and each model type has sub types as well. For example, the image below has Furniture Selected, with sub types of chairs, tables, beds, etc etc. 

There are currently 44044 models in the model database and 2236 animations in the animation database. Clicking on the tile downloads the zip or fbx file. This behavior is modified in tile_layout_wdg.py. 



Monday, November 16, 2015

Postgres Comands

General

\copyright show PostgreSQL usage and distribution terms
\g [FILE] or ; execute query (and send results to file or |pipe)
\gset [PREFIX] execute query and store results in psql variables
\h [NAME] help on syntax of SQL commands, * for all commands
\q quit psql
\watch [SEC] execute query every SEC seconds

Query Buffer

\e [FILE] [LINE] edit the query buffer (or file) with external editor
\ef [FUNCNAME [LINE]] edit function definition with external editor
\p show the contents of the query buffer
\r reset (clear) the query buffer
\s [FILE] display history or save it to file
\w FILE write query buffer to file

Input/Output

\copy ... perform SQL COPY with data stream to the client host
\echo [STRING] write string to standard output
\i FILE execute commands from file
\ir FILE as \i, but relative to location of current script
\o [FILE] send all query results to file or |pipe
\qecho [STRING] write string to query output stream (see \o)

Informational

(options: S = show system objects, + = additional detail)
\d[S+] list tables, views, and sequences
\d[S+] NAME describe table, view, sequence, or index
\da[S] [PATTERN] list aggregates
\db[+] [PATTERN] list tablespaces
\dc[S+] [PATTERN] list conversions
\dC[+] [PATTERN] list casts
\dd[S] [PATTERN] show object descriptions not displayed elsewhere
\ddp [PATTERN] list default privileges
\dD[S+] [PATTERN] list domains
\det[+] [PATTERN] list foreign tables
\des[+] [PATTERN] list foreign servers
\deu[+] [PATTERN] list user mappings
\dew[+] [PATTERN] list foreign-data wrappers
\df[antw][S+] [PATRN] list [only agg/normal/trigger/window] functions
\dF[+] [PATTERN] list text search configurations
\dFd[+] [PATTERN] list text search dictionaries
\dFp[+] [PATTERN] list text search parsers
\dFt[+] [PATTERN] list text search templates
\dg[+] [PATTERN] list roles
\di[S+] [PATTERN] list indexes
\dl list large objects, same as \lo_list
\dL[S+] [PATTERN] list procedural languages
\dm[S+] [PATTERN] list materialized views
\dn[S+] [PATTERN] list schemas
\do[S] [PATTERN] list operators
\dO[S+] [PATTERN] list collations
\dp [PATTERN] list table, view, and sequence access privileges
\drds [PATRN1 [PATRN2]] list per-database role settings
\ds[S+] [PATTERN] list sequences
\dt[S+] [PATTERN] list tables
\dT[S+] [PATTERN] list data types
\du[+] [PATTERN] list roles
\dv[S+] [PATTERN] list views
\dE[S+] [PATTERN] list foreign tables
\dx[+] [PATTERN] list extensions
\dy [PATTERN] list event triggers
\l[+] [PATTERN] list databases
\sf[+] FUNCNAME show a function's definition
\z [PATTERN] same as \dp

Formatting

\a toggle between unaligned and aligned output mode
\C [STRING] set table title, or unset if none
\f [STRING] show or set field separator for unaligned query output
\H toggle HTML output mode (currently off)
\pset [NAME [VALUE]] set table output option
(NAME := {format|border|expanded|fieldsep|fieldsep_zero|footer|null|
numericlocale|recordsep|recordsep_zero|tuples_only|title|tableattr|pager})
\t [on|off] show only rows (currently off)
\T [STRING] set HTML <table> tag attributes, or unset if none
\x [on|off|auto] toggle expanded output (currently off)

Connection

\c[onnect] {[DBNAME|- USER|- HOST|- PORT|-] | conninfo}
connect to new database (currently "postgres")
\encoding [ENCODING] show or set client encoding
\password [USERNAME] securely change the password for a user
\conninfo display information about current connection

Operating System

\cd [DIR] change the current working directory
\setenv NAME [VALUE] set or unset environment variable
\timing [on|off] toggle timing of commands (currently off)
\! [COMMAND] execute command in shell or start interactive shell

Variables

\prompt [TEXT] NAME prompt user to set internal variable
\set [NAME [VALUE]] set internal variable, or list all if no parameters
\unset NAME unset (delete) internal variable

Large Objects

\lo_export LOBOID FILE
\lo_import FILE [COMMENT]
\lo_list
\lo_unlink LOBOID large object operations