Upcoming release VoxelPipeline (Importer/Exporter)

Jan 23, 2014 at 11:35 AM
Hi all!

First, thanks for this excellent program :)

Let me introduce soon-to-be-released as a plugin and a standalone command line program to export MagicaVoxel models to 3D modelling suites/games : VoxelPipeline.

The need for this arose when I tried to export models straight from MagicaVoxel. That resulted in lots of vertices and faces as MagicaVoxel exports the voxels as cubes and we needed a way to just export all the visible surfaces.

For example voxel model chr_knight exports the following
MagicaVoxel : 2330 vertices, 1460 faces
VoxelPipeline with texture as coloring : 1611 vertices, 798 faces (could be less, more on that lower)

The algo searches all the visible faces (this includes hollow sections as well) and exports those. The colors are output as either vertex colors (per face, takes more vertices/faces) or texture (can merge "flat" color differences).

Usually this kind of algo is sloooow (one "competitor" took 2 hours to export the dragon.vox), but I'm quite pleased with the performance I achieved with VoxelPipeline (dragon.vox takes 8 minutes).

It uses custom lookups for faster face expanding and parallelism so if You have multicore system, this tool benefits from it.

Underlying libs that I use are boost and assimp for export options, this sums up to a whopping 3 megabyte program, but due to those libraries it (should be) is portable across win/mac/nix.

Now to give a bit guide what VoxelPipeline can do :
  • Export to collada dae/obj/ply/stl/stlb
  • Transform input (rotation/scale/translation), for example when exporting to collada assimp assumes Y axis to be up, one can rotate around x 90 degrees to get the correct result when reading dae
  • Offset after transform, this is used to position the model according to its bounding volume AFTER the transformation. Either source(keep what comes from the editor), center, positive and negative to all separate x,y,z. Useful when animator want's the model centered on x and y but positive z to "stand" on the modeller plane
  • Overwrite colors, if You want to force r,g,b,a channel to be something else than what it is in the editor You can. Mostly needed for default palette models, since some of the colors have invalid alpha channel.
  • Output color as vertex colors or uv texture mapping. For fastest possible output (yet it takes a bit more vertices and faces) one can export model with vertex coloring. After banging my head to the wall for hours to figure out why in Blender the vertex colors don't show up, I read that it doesn't support that. Well to combat this blender fault I implemented texture output (you can define voxel-to-pixel scale for mipmapping/lod stuff since ratio of 1 makes ugly models). The output is correct and produces good looking models but the unwrapping is naïve /slow (majority of the time is spent on finding positions on face textures).
Phew, that's about it, some notes how ever.

Will it cost money? No.
Will the results be under propriety license? No, Your models Your choice.
Will I release the source? Don't know, probably, haven't decided it yet - ephtracy/glkasumi is more than welcome to get it since this was primarily meant for MagicaVoxel.
Will it support more formats? When I have time, if people request it then sure. This is built in expansible way to support more input/output format (even model to voxel if need be).
When it is public? When I have tested all the features one more time I plan to release beta for all.

Additional note for game developers : VoxelPipeline exports all faces as "pure" faces, this means that their vertex normals are the same as face normals (flat shading). It used to have a "smooth" export function but I scrapped it in the last minute. The reason was that even though it calculated accurately the smoothed normals, it would have required a complex unwrapping algorithm to combine same vertices and plus the lighting results were bad for smooth normals (voxels aren't supposed to be rendered like that! too many artifacts!). Best way to get "smooth"/shadowed voxels is to use the MagicaVoxel way : render with flat shading, add SSAO(or similar) on top of it. However if You are feeling adventurous, here's how to do it blender (which has excelent unwrap possibilities), I presume Maya, 3ds max has similar ways :
  • Import dae (or similar) with texture
  • Create a second uv map
  • Unwrap with Your choice of algo (smart uv does good job) to the just created uv map
  • Bake textures from the original uv map to the second one
  • Delete the first one
  • Select all vertices -> remove duplicates
    This results is MUCH lower vert/tri count but also weird lighting, could suit Your needs tho.
Thanks for reading! More info on upcoming days :)
Jan 23, 2014 at 7:05 PM
This is great to hear!

I'm using this as a primary source of model creation for a game I'm building in Unity engine. It's incredibly easy to get some good looking models out there. I've noticed a few things that can be improved though. Maybe your script, or some additional configuration in the MV editor can help me, but I haven't explored any solutions yet. I was in the process of doing that when I arrived here.

If I may provide some input to you, and the creator of this wonderful program (thank you!)
Hopefully this information can help you make a robust extension, and apologies if I'm repeating what you are trying to achieve, I'm not a 3d artist by trade. ;)
  1. Option to set origin to Xmax/2, Ymax/2, Z0
    It takes me a long time to export the models, load them up in Blender, and reset origin to bottom-center for my game. I don't know if there is a better way to do this or if that has yet to be implemented. Looks like you mentioned an option to set to center, center, 0 which is great.
  2. Specify size of exported voxels - scale transform?
    It would be nice if we could specify the voxel size in config for exported objects because even 16^3 unit models appear massive on the unity scale. I suppose I could change my framework a little to accommodate for the standard size of the models, but I like my position units to represent something along the lines of 1m, not 1mm. That length scale poses a problem when the raw exported model appears to be 6m^3.
  3. Rounded or Beveled Voxel corners
    I like this idea from another discussion thread. It gives a great look. Maybe it would make a nice feature to your exporter.
    https://voxel.codeplex.com/discussions/474648
  4. Bake materials into exported models
    This would be nice but it is not a priority for me now, I am happy you are implementing something like this.
    I do like that the models can all be mapped to one texture which helps my game performance-wise. Additionally, I still have yet to find a way to color my models programatically, though I suppose that is more of a Unity question. I'm planning on having an RGB value for models that will be generated based on conditions, don't know if any kind of extension or change in logic can help provide for this.
Thanks again to all who have contributed to this project.
Coordinator
Jan 24, 2014 at 1:32 AM
Edited Jan 24, 2014 at 2:52 AM
Hi Dcl2049,

Thanks for your suggestions.

Will add them to the todo list:)

and :
  1. in the new release, the pivot of export is on the center of bottom for .obj, .3ds,and .ply
  2. for color information, .obj and .3ds already have texture coordinates encoded.
while .ply has vertex color encoded
Jan 24, 2014 at 9:21 AM
Hi Dcl2049,
  1. This is already supported, --output-offset-x=center --output-offset-y=center --output-offset-z=positive will do what You want.
  2. This also is supported, --output-scale-x=100 --output-scale-y=100 --output-scale-z=100 will make blender see the one voxel as 1 unit (the default for assimp is 0.01 units per coordinate).
  3. This is a neat feature, but I think I'll leave this to blender, since it's kind of easy to there with a script, however, You should be aware that vert/tri count will sky rocket if You do this :) For example the chr_knight is already 798 triangles and it's somewhat simple model. Now if You "extrude" one voxel to make it rounded, it will make the original 2 triangle face to ... calculating ... 10 triangles minimum. This would make a whopping 7980 triangles for a single character. If You aim for mobile games, it could kill Your fps.
  4. Yes diffuse texture baking is already done, so all the colors will be in texture after export. If You want a better solution that isn't texture resolution dependant You should use vertex colors. As a former Unity user it's easy to just make a shader that uses the vertex color information to render the diffuse (I think there could be even a standard out-of-the-box shader that does this). However note one thing, if Your workflow is MagicaVoxel -> Blender -> Unity, be sure to use ply files for vertex color output since AFAIK it is the only supported format in blender that reads vertex colors. So if You use collada for example You will lose the vertex colors on blender import.
Jan 24, 2014 at 9:42 AM
Edited Jan 24, 2014 at 9:42 AM
As for example, the following line :
VoxelPipeline.exe --input-path=test.vox --input-format=magicavoxel150 --output-path=test.dae --output-format=collada --output-color-type=texture --output-texture-scale=10 --output-scale-x=10 --output-scale-y=10 --output-scale-z=10 --output-rotation-x=90 --output-offset-type-x=center --output-offset-type-y=positive --output-offset-type-z=center

Will produce the following setup when opened in blender (sorry for the quality of the pic) :
Image

A little explanation is in order for the parameters : we need to rotate around x axis 90 degrees since assimp insists that the up vector is Y_UP and in blender and MV it's Z_UP. We scale the output texture to scale 1 voxel = 10 pixels, to get the desired quality of the texture, other options are basic stuff :)
Jan 25, 2014 at 7:54 PM
Edited Jan 25, 2014 at 7:56 PM
Hi, just a little update, testing is going fine, tomorrow maybe I'll release an alpha for MagicaVoxel users (note, not the plugin version yet, the standalone version).

I added a neat little feature called "texturecombine" for color output. If You don't care about the diffuse texture (You just care that the model looks the same way that in MagicaVoxel), and don't plain to ever use modify the diffuse/uv texture by hand, it is possible to output the colors via texture that reuses common colors. This makes a HUGE difference on some models that have lots of same colored faces. It works by creating just 1 pixel size plan color and wrapping UV around that point, if there are other same colors it reuses the one pixel output created before.

For example, the dragon.vox (that is actually hollow, I presume glkasumi imported it from obj), outputs an insane 4096x4096 texture for all the faces. Fair enough, if you want to draw on those faces, You can. Now if using the "texturecombine" method, it outputs 1x1 texture. The difference? TGA 4096x4096 takes 65 megabytes, the 1x1 takes couple of bytes.

Hope You like it, it will be the last feature addition before alpha/beta release :)
Jan 26, 2014 at 8:59 PM
Thanks guys for the input. The software is awesome and I am still learning a ton about creating 3d objects for my game, so it's definitely a learning process.

The software in its current state is really awesome but I feel like for all the ID-10-T users out there (me included :D) it could benefit from some UI changes/additions and some more user friendly options prompts.
Coordinator
Jan 27, 2014 at 3:19 PM
Edited Jan 28, 2014 at 11:11 PM
Hi Saviilsy,

Appreciate it! Looking forward to the release:)

Hi Dci2049,

Thank you :) you can start new discussion thread for any suggestion or new idea,

and it may be added as a feature in the future when everything is ready :)
Jan 28, 2014 at 5:15 PM
Hi!

Ok, You can download the 0.9 beta from http://sdrv.ms/1aFetKA

Report any problems here, I'll do a proper "product" page on my own web site when I have time.

Have phun!
Coordinator
Jan 28, 2014 at 11:12 PM
Nice! I'll try it later:)
Coordinator
Jan 29, 2014 at 9:17 PM
Edited Jan 29, 2014 at 9:18 PM
I think I need to add query methods and some config setup to simplify the command line input work:)

and the option is --input-file not --input-path?
Jan 30, 2014 at 5:49 AM
Yea, sorry. I refactored a bit before releasing. It SHOULD show correct arguments with --help, if not, please tell me :) But --input-file and --output-file.

There is an option to use external file with "--options-file". With this one can make a "base template" for conversion. This feature is heavily used in the plugin version, as You can make "VoxelPipelineOptions.txt" and make all the "basic" setup there, then just let the MagicaVoxel/Plugin handle the rest. The workflow is actually really nice when set up.

The reason I asked you about getting the runtime dir of MV is to get the plugins dir. There I could store that options text file and log.
Jan 30, 2014 at 7:14 AM
Hi, how do i use this plugin?
Jan 30, 2014 at 7:32 AM
Hi inf1n1tyIt,

This is not the plugin version yet, its the standalone version. So You need to open Command Line and use it from there.

Hopefully the plugin version will be ready in the weekend, work has already been started and it's going fine.
Feb 4, 2014 at 11:36 AM
Edited Feb 5, 2014 at 11:03 AM
Hello I really liked your programa. I love it now. Prompt out of it once in Eeksortirovat Unity3d with color.
Feb 6, 2014 at 4:30 PM
Can you explain where do i put this and how do i use it?
Coordinator
Feb 7, 2014 at 6:19 AM
Hi Crashic,

Thanks:) The default exported model has texture coordinates encoded. The texture used by the model is the color palette which can also be exported as .tga or .act
Coordinator
Feb 7, 2014 at 6:24 AM
Hi inf1n1tylt,

This one that saviilsy developed is a standalone version which needs external command line.

When the plugin version comes out, it can be accessed from the editor ( and may also save some of the typing work).

I will write some simple ones to demonstrate how to use plugin in editor. Thanks.
Feb 8, 2014 at 9:01 PM
Hi glkasumi. Prompt everything such as how to achieve such a result in your editor, "Shadows on the corners models" or as it's called?
Coordinator
Feb 10, 2014 at 10:52 PM
Edited Feb 11, 2014 at 3:43 PM
Hi Crashic,

It is called "ambient occlusion", or ao in short.

Basically it means how much light intensity a point can receive from the sky in its positive hemisphere.

The simplest way to implement this to to calculate the ao at each vertex.
Feb 15, 2014 at 11:58 AM
This sounds great, looking forward to the plugin version.
Feb 16, 2014 at 11:36 AM
Hi!

Sorry for the slight quietness, my day job took all of the time.

I've decided the rewrite the whole thing for many reasons : assimp adds lots of unnecessary things to the pipeline and quite frankly it's kinda crap when it comes to exporting models (granted, they are IMPORT pipeline, not export). Dynamic runtimes adds bloat and I don't want to users to start thinking where to put which dll in order to work. So here is the sum that will be changed and will not :
  • Static runtime, Only one exe/dll
  • Possibility to change texture type at export
  • Possibility to embed texture into model file (if it supports it)
  • All the main algos are the same, only a bit optimized versions
  • More modular design, easy to add new features
  • Able to change handedness when export (from right to left or the opposite)
  • More parallelism, adds performance
  • Only collada export supported at release (obj to follow short after, as well as qubicle constructor import)
  • Name changed to VoxelPolygonizer :)
It's about 80% done at the moment, but You can use the above command line version for now, results will be the same when using this new version.
Mar 6, 2014 at 4:29 AM
Edited Mar 6, 2014 at 5:24 AM
I am waiting for this.
so hard right now.

What kind of formats will this support (better question would be what formats would this support and still maintain color information?) and how does one go about applying that color information to the model? (in as much detail as you can provide for me and anyone else looking for an answer). Do i have to apply it manually through the "default.tga" or does it get imported when i import the model into blender/unity/maya?

thanks.

keep up the good work!
Mar 11, 2014 at 4:37 PM
Lurked here for a bit- decided to sign up at codeplex just to be able to write to you personally saviilsy.

First of all I want to say I am super pumped about this plugin and what it can do, and is a huge factor into why I am looking into using MagicaVoxel as an art authoring tool for the voxel game I am making. This plugin is awesome- you by extension are awesome. :P

I know you're almost done rewriting the plugin again- but I wanted to offer some feedback (I don't know if it is too late?!)

Notably about how the UVs are generated when using the texture option in your first build you released- I realized this might've changed. But I noticed that when the UVs are generated, the UV points appear to be placed in the center of the pixels, as opposed to at the corners of the pixels. Snapping the points to the corners of the pixels would be what I would consider to be the correct way to do things, as the pixels between the different UV islands would line up and the voxel look would appear intact.

I'm not sure if I'm explaining myself right- if that sound confusing let me know and I can provide further explanation or pictures.

I'm not going to pretend I know anything about how to code something like a UV mapper- but I imagine it would involve using determined texture resolution (n) to divide up the 0-1 UV space, subtracting the modulo of the resulting divisions, and piping that final value to the UV points themselves in order to get them to snap the corners.

Again I wanna say thank you for making such a useful plugin! This makes MagicaVoxel just THAT much better! But in my opinion its suffering from this one little thing- and if you have addressed it, it would be perfect!

Let me know what you think!
Mar 18, 2014 at 3:43 PM
Thought I'd share this.

My buddy made a front end for this. Simple stuff, but removes need to use command prompt. :P

https://dl.dropboxusercontent.com/u/17002839/VoxelPipeline%20FrontEnd.exe
Coordinator
Mar 20, 2014 at 6:54 PM
Hi Rooster128,

Thanks! this front end works nicely.

By the way, it seems that the translation and rotation doesn't accept negative values.
Mar 21, 2014 at 3:33 AM
Thanks for the feedback! (Friend thanks you too!)

Update with fix:

https://www.dropbox.com/s/hm0k2enh99nr7qa/VoxelPipeline%20FrontEnd.exe

PS:

http://24hoursofhappiness.com/
Mar 29, 2014 at 12:26 AM
hey Rooster, i downloaded your front end but once again cannot figure out how to get it to work, i ahve put in the voxelpipeline location, the .vox location, the output file set up my texture output whatevers...
how am i supposed to get it to work...or launch from there...or convert or whatever other word i should use? hahaha, sorry im fried today and cant come up with a proper way of asking you this.

theres no "go" button or anything anywhere and i am confused.
Mar 29, 2014 at 7:22 PM
It's a rather big window- you may need to move the window past the top of your screen. At the bottom of the window is an "Export" button.
Mar 30, 2014 at 10:58 PM
Thanks for the reply dude.
That definitely seems to be the case. I noticed the box at the bottom of the front end extended further than the window.
I have tried to run it using shortcut resolution commands and stuff and can't reach it (I'm on a laptop so I'm running 1366x768 resolution).
So I'm assuming you guys made it to run on a 1920x1080 monitor.
Is there anyway to run the program at a different resolution? Or is there any sort of file I can edit to change the location of the button.
Anything like that?
I'm sorry but this is the one program I absolutely need as this point to work and I have had zero luck with command line and I was hoping that front end would help me out.
Unfortunately I can't reach the button hahaha. I can't even tab down to it.
Apr 4, 2014 at 5:03 PM
Told me buddy about this. He's been busy with UE4 stuff but he said he'd look at it after.

saviilsy are you still around?
Apr 13, 2014 at 4:50 PM
Model optimization worked great! I wasn't able to get the textures working though, the tga looks like it'd work but the UV map is messed up. Is there a trick to it?
Apr 22, 2014 at 12:20 AM
Edited Apr 22, 2014 at 12:23 AM
cortlong50 wrote:
Thanks for the reply dude.
That definitely seems to be the case. I noticed the box at the bottom of the front end extended further than the window.
I have tried to run it using shortcut resolution commands and stuff and can't reach it (I'm on a laptop so I'm running 1366x768 resolution).
So I'm assuming you guys made it to run on a 1920x1080 monitor.
Is there anyway to run the program at a different resolution? Or is there any sort of file I can edit to change the location of the button.
Anything like that?
I'm sorry but this is the one program I absolutely need as this point to work and I have had zero luck with command line and I was hoping that front end would help me out.
Unfortunately I can't reach the button hahaha. I can't even tab down to it.
Got some time to update the frontend, and to make a codeplex account, for the old version if you selected the VoxelPipeline Location textbox, so you could type in it, and then pressed tab twice, you would have had the Export button selected, then pressing space or enter would have simulated a click on it. Here is a link to the newly updated version (same link as the last one but here it is again if you don't feel like scrolling up) https://www.dropbox.com/s/hm0k2enh99nr7qa/VoxelPipeline%20FrontEnd.exe , if you can not see the Export button, if you double click on a blank space on the application, the command line box, which is normally right above the Export button, will be removed and the application form will shrink to a height of 661 pixels, which should be small enough to fully fit on anyone's screen that is using Magicapoxel and wants to use the VoxelPipeline with a frontend. If you want to have the command line box back again, just double click in a blank spot on the application form again, and the command line box will be added back and the form will grow again to its normal height of 819 pixels. Any other questions, comments or feedback for the frontend, feel free to post them here.


Also I'm looking forward to the new update to VoxelPipeline/VoxelPolygonizer, Saviilsy
Apr 26, 2014 at 1:47 PM
Hey guys, first off I want to thank Saviilsy and glkasumi a ton.

And then secondly I want to ask if there was any documentation on getting the textures to work correctly ? I cant seem to get what I have in MV but the mesh itself looks ok.
May 1, 2014 at 12:27 AM
Edited May 1, 2014 at 1:28 AM
Xennma, thanks a ton dude. This on actually lets me export....the jury is out on whether or not its going to work, which i will come back and let you know asap.

Thanks a ton, if it works the way i hope you will have just started an indie dev on his journey to making a game haha.

thanks man.


okay, so i have got it to export. Now! the question is....how do i get it to properly output the texture and how would i open this model (textured) into blender or unity engine?

Im sorry, but i have messed with out texture parameters and im really just confused.




EDIT AGAIN! SORRY! hahaha, okay i got it and im so stoked i crapped. i did.

heres an image along with the settings i used. (sorry for the quality)

[IMG]http://i.imgur.com/XBCzzLB.png[/IMG]




here are the settings.

C:\Users\NOMI SUNRIDER\Documents\3d editing\MagicaVoxel\export\VoxelPipeline.exe --input-file="C:\Users\NOMI SUNRIDER\Documents\3d editing\MagicaVoxel\vox\m4-3-2.vox" --input-format=magicavoxel150 --output-file="C:\Users\NOMI SUNRIDER\Documents\3d editing\MagicaVoxel\vox\m4.dae" --output-format=collada --output-color-type=texture --output-texture-scale=10 --output-color-red=0 --output-color-green=0 --output-color-blue=0 --output-color-alpha=0 --output-offset-type-x=center --output-offset-type-y=positive --output-offset-type-z=center --output-scale-x=10 --output-scale-y=10 --output-scale-z=10 --output-translation-x=0 --output-translation-y=0 --output-translation-z=0 --output-rotation-x=90 --output-rotation-y=0 --output-rotation-z=0
May 5, 2014 at 12:19 AM
Just a quick update:

I took over the work from saviilsy and integrated a collada exporter into VoxelShop (http://blackflux.com/node/11).

The exporter has several options. Most importantly a T-Junction save option and a low triangle option (see here for the theory: http://blackflux.wordpress.com/2014/02/23/meshing-in-voxel-engines-part-1/). It also does do very intelligent texture compression. Work will continue, but it's already very useful and easy to use.

Looking forward to your feedback!
Coordinator
May 6, 2014 at 2:40 AM
Hi cortlong50,

Great artwork for the weapons!

Hi Flux,

Thanks for taking over the work and improving it:)
Jun 22, 2014 at 7:32 PM
Thank you! This frontend has been the single most important thing between making models and gettin them into the engine, without it my game wouldnt be under development. So thank you so much. more help than you will know!

Ill leave you with 2 more samples haha.

http://imgur.com/8e35vtq

http://imgur.com/dnLOMyT
Jul 2, 2016 at 2:51 AM
IMPORTANT UPDATE:
VoxelShop is now open source! Always looking for contributors and feedback! If you have any issues or suggestions please post them here: https://github.com/simlu/voxelshop/issues

I'm not monitoring this thread regularly. Github is your best bet if you want a timely response!