Category Archives: 3D Reality Capture

Think Before You 3D Scan That Building

There is a growing awareness and understanding of the intellectual property considerations in the capture/modify/make ecosystem – particularly as it relates to content that is captured via a 3D scanner, modified or mashed up, and then manufactured (via 3D printing or otherwise).   I have written about this before – blogging early in 2012 in The Storm Clouds on the Horizon that I felt the next “Napster” era was upon us for digitally captured real world content.   In the last two years there have been transformative technical changes on both “ends” of that ecosystem, consumer/prosumer 3D printing solutions along with an emerging class of inexpensive 3D real world capture devices and software solutions.

Earlier in 2014 I identified the following key trends in the capture/modify/make ecosystem for object based 3D capture and reproduction:

2014 Market Trends

Intellectual Property Concerns in Scene Based 3D Capture?

I’d point you to the on-going discussion around the metes and bounds of intellectual property protection in the object based capture/modify/make ecosystem is interesting context for some high level issue spotting with the intersection of intellectual property and scene/world based reality capture (as part of a formal digital documentation process or as part of an informal, crowd-sourced creation of a 3D world model).   What you’ll find is a lot of gray areas, inconsistencies, and mind melting information.

Understanding intellectual property in the context of scene based scanning will become more relevant over the coming years for many of the same reasons that have driven change and awareness in the context of capturing and reproducing objects.

The falling cost of high accuracy scanners useful for scene based 3D capture (e.g. the FARO Focus3D family means that more data will be captured, by more users, in the commercial context (either directly by the owner/operators themselves or by third party scanning service bureaus).  Similarly, 3D data capture will become ubiquitous on the consumer side later in 2014 and beyond – as Intel adds their RealSenseTM depth sense technology to every laptop they ship, Google progresses with Project Tango along with their software partners and as lightfield camera technology goes mainstream.  In my opinion, we are not too far away from the creation and continuous update of a 3D “world model” populated with data coming from various types of consumer, professional and industrial sensors.

Intellectual Property Applies to Buildings and Public Spaces?

I hate to be the bearer of bad news, but to answer the question, “yes” intellectual property rights may be impacted if you capture 3D data of a building, public spaces or other real world scenes. That’s probably surprising to many of you.  For purposes of this discussion I’m only going to focus on how the laws in the United States might apply (and in many respects they differ from other countries around the world – for an example of just how interesting/different that can be, see:  Your mileage can and will vary, this is presented to help with issue spotting – and not to provide definitive guidance on any particular situation.

I had the opportunity to speak on this topic with Michael Weinberg from Public Knowledge at a FARO Digital Documentation Conference a few years ago – many of the issues and examples we discussed then are still relevant now (and even more so with the explosion of low cost 3D capture solutions).

You Can Infringe Intellectual Property By Scanning A [Building] [Lobby] [Plaza] [etc.]?

The intellectual property ramifications of capturing 3D data in the context of a scene is very muddled.  Very little case law addresses these issues, and that which does, isn’t very clear.  Naturally, most buildings would not be copyrightable because, by their very nature, they are “useful” – and something that is useful is generally not afforded copyright protection (yes, useful objects can be patented, and you can actually patent certain building elements – but that’s a topic for a different day).

But think about something like a sculptural memorial that is in a public plaza.  Would the sculptor of the memorial be afforded copyright protection?  You betcha, as a sculptural work is specifically protected under the United States Copyright Act.   What if you were to take a picture of that memorial and decide to license it to the United States Postal Service for use on a postage stamp.  Would that picture, and re-use, require you to obtain clearance (in the form of a license) from the sculptor before you could do so, and before the USPS could sell the stamps?
Korean War Veterans Memorial Sculpture

According to the 5th Circuit Court of Appeals, when they looked at this question in 2010 – the answer was “yes.”  The failure to get a license from the sculptor, even though the defendant obtained one from the general contractor which installed the sculpture, constituted copyright infringement.   After remand back to the United States Court of Federal Claims, and a subsequent appeal, the sculptor, Frank Gaylord, was awarded $685,000 in the fall of 2013.

So what if you plopped your scanner in the middle of that field and captured a 3D point cloud (that mere act is likely infringing)?   What if you decided to sell that data to a third party and then printed 3D prints from the data?  What if you used that data as part of an immersive augmented reality platform to promote tourism for Washington DC?   Would you/could you be liable?

OK Sculptures Maybe, But A Building?

Empire State Building

A building is the essence of utilitarian and functional, so we are safe from copyright, right?  You might think so.  But you’d be wrong.

Cooper Union Building

Take for example the above, which is the Cooper Union New Academic Building in New York City. Construction was finished in 2009.   Still utilitarian and functional (umm, as a building) so not copyrightable, right?

Wrong. In the United States, under the Architectural Works Copyright Act of 1990 a building designed (and that design is fixed in a tangible medium – i.e. drawings, or actually constructed) after 1990 is specifically subject to copyright protection (although purely functional or utilitarian aspects of a building are not protected).  So, if you set up your 3D scanner on the sidewalk here and captured a point cloud of the Cooper Union New Academic Building, have you committed an act of copyright infringement?

Alright how about this awesome bridge –


It’s a bridge, so the essence of utilitarian and functional.  It also has significant artistic and sculptural elements. Since it’s not intended for human habitation, the Architectural Works Copyright Act does not apply.  Phew.

How about this pavilion built at the Fort Maurepaus beach park, located in Ocean Springs, Mississippi and constructed by FEMA after Hurricane Kartrina?


Folks aren’t supposed to live in pavilions, so we are safe, right?  Wrong.  Pavilions are specifically covered by the Architectural Works Copyright Act.  Only if it was built after 1990.  And it was. Oye.

 But Wait, We Have An Exception!

There is an exception to acts of infringement under the Architectural Works Copyright Act –

 The copyright in an architectural work that has been constructed does not include the right to prevent the making, distributing, or public display of pictures, paintings, photographs, or other pictorial representations of the work, if the building in which the work is embodied is located in or ordinarily visible from a public place.

United States Copyright Act, 17 U.S.C. Section 120 (as amended).

Yippee, so if we user our 3D scanner to capture a point cloud of a building we are saved by the so-called “photographer’s exception” for buildings that have been constructed and can be seen from a public place.  Right?  Unfortunately it’s a big “I have no idea but don’t think so but couldn’t find any case law that answers this question”.

Baltimore Train Station

The above is a photo of Baltimore Penn Station.  This was certainly built before 1990 – so no protection under the AWCA.   Even if it was built after 1990 – we would still OK because it would be covered by the photographer’s exception (assuming that applied to 3D data acquisition), right?

Wrong.  Even if that exception were extended to 3D data capture via scanning, that specific statutory exception does not apply to sculptures (or other objects protected by copyright) which are separable from a building.  [FWIW, the sculpture is called Male/Female by Jonathan Borofsky].

Other countries (e.g. Canada, Ireland, the UK) extend the photographer exception concept to all publicly located, but otherwise copyrightable, works (e.g. sculptures).  Not in the United States though.

Think You Are Confused Now?

What about a sculpture in a building?

Sculpture Attached to a Building

Or how about a sculpture that is attached to a building?

Sculpture in a Building

Or what about a building built by Yale in 2005, CALLED the “Sculpture Building” –

The Sculpture Building

Or how about a building that IS a sculpture (like the Walt Disney Concert Hall in Los Angeles, designed by Frank Gehry?)

Frank Gehry Concert Hall

Policy Implications of Ubiquitous 3D Data Capture of Scenes

In addition to the various intellectual property concerns that are potentially touched about when 3D scenes are captured, I believe there a host of other privacy and ownership issues that need to be thought through as well.  If I’m a facility owner, and I don’t want data captured – how do I prevent it as devices become more ubiquitous?  Sure, I can require people to leave their phones at the security desk (many secure facilities already have no photography or data transfer processes), but what do I do about their glasses?   If I’m a contributor of 3D data to a community sourced 3D “world model” who owns the data that I capture and upload?   Who is responsible if it is ultimately found to be infringing?   What are the policy and legal implications if Google, instead of capturing photographs for their street maps, instead, created 3D point clouds of every place they went?

Some Practical Advice for Service Providers

So what can you do minimize your risks if you are a commercial scanning service provider and are engaged to do some scene based scanning?

  • Ask questions —  Know enough to generally understand the potential risks and pitfalls of any data capture engagement.
  • Transfer liability and responsibility for clearance  – As a service provider, make sure that the owner/operator or the entity which engaged you to do complete the work is responsible for intellectual property clearance issues and agree to hold you harmless (e.g. they are responsible, not you, for any potential infringements).
  • Be especially careful with artistic elements –  Creative and sculptural elements should be subject to more scrutiny.  For example, if you are asked to scan a building lobby, and there is a sculpture in the middle of it, you should specifically get clearance from the artist.
  • Know how the collected data will be used —  Be absolutely clear on the data ownership and the plans for downstream use.  Is the data going to be used as part of a digital documentation process (so no broad public dissemination) or is going to be published and made accessible as part of an augmented reality application?

About the Author

Tom Kurke was the former President and Chief Operating Officer of Geomagic, a specialist supplier of 3D reconstruction and interaction software and hardware solutions – which was acquired by 3D Systems Corporation (NYSE: DDD) earlier this year.  Prior to Geomagic he spent more than a decade with Bentley Systems, a leading providing of solutions to the designers, constructors owners and operators of some of the largest constructed assets in the world.  He recently joined the Board of Advisors of Paracosm ( whose mission is to “3D-ify the World.”  When not supporting his two sons various sporting activities, or writing on topics of interest in the areas of 3D printing, digital reality capture, intellectual property, AEC/GIS or unmanned aerial systems at, you might see him finding new ways to crash his quadcopter.

[Note: This article was originally published on LiDAR News on April 26th, 2014, you can find that here.]

LazeeEye – 3D Capture Device Phone Add-On

There has been a continuing strong push on the consumer/prosumer 3D reality capture side of the capture/modify/make ecosystem – whether that captured content is to be used in an object or scene based scanning workflow.  New processing algorithms along with orders of magnitude improvement in processing power are unlocking new capabilities.

DIY scanning solutions have been around for a while – ranging from pure photogrammetric approaches, to building structured light/laser scanning setups (e.g. see the recommendations which DAVID 3D Solutions GbR makes on the selection of scanning hardware, by leveraging commercial depth sense cameras in interesting new ways (e.g. leveraging PrimeSense, SoftKinetic or other devices to create a 3D depth map or by utilizing light field cameras for 3D reconstructions . Occipital raised $1M in their Kickstarter campaign to develop their Structure Sensor (which is powered by PrimeSense technology) hardware attachment for Apple devices and 3D Systems is white labeling that solution.   Google has been working on Google Tango with its project partners (and apparently Apple – because the Google Tango prototype included PrimeSense technology)!

Early in 2014 I looked at the various market trends that were impacting the capture/modify/make ecosystem — the explosion of low cost, easy to use 3D reality capture devices (and associated software solution stack and hardware processing platforms) were part of the key among them –

2014 Market Trends

For a graphical evolution of how some of the lower cost sensors have developed over time, see:

3D Sensor Progression

Along comes an interesting Kickstarter project from Heuristic Labs for the LazeeEye, which so far has raised roughly $67K (on a goal of $250K) to develop a laser emitter which attaches to a phone, which flashes a pattern of light onto the object or scene to be capture, and stereo vision processing software on the phone creates/infers a depth map from that.  According to Heuristic Labs, the creators of the LazeeEye:

LazeeEye? Seriously? The name “LazeeEye” is a portmanteau of “laser” and “eye,” indicating that your phone’s camera (a single “eye”) is being augmented with a second, “laser eye” – thus bestowing depth perception via stereo vision, i.e., letting your smartphone camera see in 3D just like you can!

The examples provided in the funding video are pretty rough, and because it is a “single shot” solution, only those surfaces which can be seen from the camera viewpoint are captured.  In order to capture full scene, multiple shots would need to be captured, registered and then stitched together.   This is not a problem that is unique to this solution (it is a known element of “single shot” solutions).  More from the LazeeEye Kickstarter project pages:

How does LazeeEye work? The enabling technology behind LazeeEye is active stereo vision, where (by analogy with human stereo vision) one “eye” is your existing smartphone camera and passively receives incoming light, while the other “eye” actively projects light outwards onto the scene, where it bounces back to the passive eye. The projected light is patterned in a way that is known and pre-calibrated in the smartphone; after snapping a photo, the stereo vision software on the phone can cross-reference this image with its pre-calibrated reference image. After finding feature matches between the current and reference image, the algorithm essentially triangulates to compute an estimate of the depth. It performs this operation for each pixel, ultimately yielding a high-resolution depth image that matches pixel-for-pixel with the standard 2D color image (equivalently, this can be considered a colored 3D point cloud). Note that LazeeEye also performs certain temporal modulation “magic” (the details of which we’re carefully guarding as a competitive advantage) that boosts the observed signal-to-noise ratio, allowing the projected pattern to appear much brighter against the background.

Note that a more in-depth treatment of active stereo vision can be found in the literature: e.g., and

[Side note, I found it interesting that Heuristic Labs is using Sketchfab to host its 3D models – yet another 3D content developer/provider who is leveraging this great technical solution for 3D content sharing.]

Depending on the funding level you select during the campaign you get different hardware – varying laser colors (which impact the scan quality), whether it is aligned, SDK access, etc.  They readily acknowledge that 3D capture technologies will become more ubiquitous in the coming years with the next generations of smartphones (whether powered by active technology like the PrimeSense solutions or passive solutions such as light field cameras) – their answer – why wait (and even if you wanted to wait, their solution is more cost effective).

Why wait indeed.  Interesting application of existing technical solutions packaged in a cheap approachable package for a DIY consumer, will be curious to see how this campaign finishes up.

[Second side note, I guess my idea of hacking the newest generator of video cameras, with built in DLP projectors (like those Sony makes), to create a structured light video solution is worthwhile pursuing.  The concept?  Use the onboard projector to emit patterns of structured light, capture that using the onboard CCD, process on a laptop, in the cloud, on your camera, etc.  Wala, a cheap 3D capture device that you take with you on your next vacation.  Heck, you are going to do that, why not just mount a DLP pico projector directly to your phone and do the same thing. . .  ;-)]


3D Printing Talk at UNCW CIE

I was fortunate yesterday to spend some time with a great crowd at the UNCW Center for Innovation and Entrepreneurship to talk about 3D Printing — sharing the time with an awesome team of presenters from GE Hitachi Nuclear Energy.  Jim Roberts, the Director of the UNCW CIE, a friend of mine since moving to North Carolina, invited me to see his impressive incubator space located at the edge of the UNC Wilmington campus – and I was glad to do so.  He has an impressive facility, and some great partner/tenant companies already working hard, I am excited to see what will be “hatched” under Jim’s leadership.  While there I also had the chance to meet with some great local entrepreneurs as well as spending some time with the Wired Wizard Robotics Team — and incredibly impressive group of young, talented, future scientists, engineers, technologists and mathematicians.   They were planning how to integrate 3D printing into their next design, I came away again believing how much STEM and the entire “capture to make” ecosystem should be intertwined.

One of the things I talked about yesterday was the interesting correlation between the performance of the publicly traded 3D printing companies and the relative rise of “3D Printing” as opposed to the technical term of “additive manufacturing”.  The upper left inserted graph is a Google Trends chart showing those search terms over time — if you haven’t used Google Trends — this data is normalized relative to all search volume over time.   In other words, a flat line would show that as a % of overall search, that term has stayed consistent (even as volume grows).  What you can see from this graph is the explosion of the rise of “3D Printing” as opposed to small, incremental growth of “additive manufacturing.”  Compare the rise of “3D Printing” to the stock charts and you see an interesting correlation indeed.  During the rest of my time I gave some reasons for why I believed this happened — looking at the macro level trends on both “sides” of the content to make ecosystem that may have unlocked this opportunity.

3D Printing + Additive Manufacturing

For those who have interest, you can download the slides I delivered here. TMK Presentation for UNCW on 3D Printing Opportunity (1.17.14 – FOR DISTRIBUTION)

Have a great weekend!

Light Field Cameras for 3D Imaging

Thanks for reading Part I of this article published at LiDAR News.  Below I examine some of the plenoptic technology providers as well as provide some predictions about 3D imaging in 2014 and beyond.  If you have been directed here from LiDAR News certainly skip ahead to the section starting with Technology Providers below.  Happy Holidays!

Light Field Cameras for 3D Capture and Reconstruction

Plenoptic cameras, or light field cameras, use an array of individual lenses (a microlens) to capture 4D light field about a scene.   This lens arrangement means that multiple light rays can be associated to each sensor pixel and synthetic cameras (created via software) can then process that information.

Phew, that’s a mouthful, right?  It’s actually easier to visualize –

Raytrix Plenoptic Camera Example

Image from Raytrix GmbH Presentation delivered at NVIDIA GTC 2012

This light field information can be used to help solve various computer vision challenges – for example, allowing images to be refocused after they are taken, to substantially improve low light performance with an acceptable signal to noise ratio or even to create a 3D depth map of a scene.   Of course the plenoptic approach is not restricted to single images, plenoptic “video” cameras (with a corresponding increase in data captured) have been developed as well.

The underlying algorithms and concepts behind a plenoptic camera have been around for quite some time.   A great technical backgrounder on this technology can be found in Dr. Ren Ng’s 2005 Stanford publication titled Light Field Photography with a Hand-Held Plenoptic Camera.   He reviews the (then) current state of the art before proposing his solution targeted at synthetic image formation.  Dr. Ng ultimately went on to commercialize his research by founding Lytro, which I discuss later.    Another useful backgrounder is the technical presentation prepared by Raytrix (profiled below) and delivered at the NVIDIA GPU Technology Conference 2012.

In late 2010 at the NVIDIA GPU Conference, Adobe demonstrated a plenoptic camera system (hardware and software) they had been working on – while dated, it is a useful video to watch as it explains both the hardware and software technologies involved with light field imaging as well as the computing horsepower required.  Finally, another interesting source of information and recent news on developments in the light field technology space can be found at the Light Field Forum.

Light field cameras have only become truly practical because of advances in lens and sensor manufacturing techniques coupled with the massive computational horsepower unlocked by GPU compute based solutions.  To me, light field cameras represent a very interesting step in the evolution of digital imaging – which until now – has really been focused on improving what had been a typical analog workflow.

Light Field Cameras and 3D Reconstructions

 Much of the recent marketing around the potential of plenoptic synthetic cameras focuses on the ability of a consumer to interact and share images in an entirely different fashion (i.e. changing the focal point of a captured scene).  While that is certainly interesting in its own right, I am personally much more excited about the potential of extracting depth map information from light field cameras, and then using that depth map to create 3D surface reconstructions.

Pelican Imaging (profiled below) recently published a paper at SIGGRAPH Asia 2013 detailing exactly that — the creation of a depth map, which was then surfaced, using their own plenoptic hardware and software solution called the PiCam.  This paper is published in full at the Pelican Imaging site, see especially pages 10-12.

There is a lot of on-going research in this space, some use traditional stereo imaging methods acting upon the data generated from the plenoptic lens array but others use entirely different technical approaches for depth map extraction.   A very interesting recent paper presented at ICCV 2013 in early December 2013 titled Depth from Combining Defocus and Correspondence Using Light Field Cameras and authored by researchers from the University of California, Berkley and Adobe proposes a novel method for extracting depth data from light field cameras by combining two methods of depth estimation.  The authors of this paper have made available their sample code and representative examples and note in the Introduction:

 The images in this paper were captured from a single passive shot of the $400 consumer Lytro camera in different scenarios, such as high ISO, outdoors and indoors. Most other methods for depth acquisition are not as versatile or too expensive and difficult for ordinary users; even the Kinect is an active sensor that does not work outdoors. Thus, we believe our paper takes a step towards democratizing creation of depth maps and 3D content for a range of real-world scenes.

Technology Providers

Let’s take a look at a non-exhaustive list of light field technology manufacturers – this is no way complete, nor does it even attempt to cover all of the handset manufacturers and others who are incorporating plenoptic technologies – nor those who are developing “proxy” solutions to replicate some of the functionalities which true plenoptic solutions offer (e.g. Nokia’s ReFocus app software).  Apple recently entered the fray of plenoptic technologies when it was reported in late November that it had been granted a range of patents (originally filed in 2011) covering a “hybrid” light field camera setup which can be switched traditional and plenoptic imaging.


Lytro (@Lytro) was founded in 2010 by Dr. Ren Ng, building on research he started at Stanford in 2004.  Lytro has raised a total of $90M with an original $50M round in mid-2011 from Andreesen Horowitz ((@a16z, @cdixon), NEA (@NEAVC), Greylock (@GreylockVC), and a new $40M round adding North Bridge Venture Partners (@North_Bridge).   In early 2012 Lytro begin shipping its consumer focused light field camera system, later in that year Dr. Ng stepped down as CEO (he remains the Chairman), with the current CEO, Jason Rosenthal, joining in March 2013.

Lytro camera inside

Inside the Lytro Camera from Lytro

I would suspect that Lytro is pivoting from focusing purely on a consumer camera to instead the development of an imaging platform and infrastructure stack (including cloud services for interaction) that it, along with third party developers, can leverage.  This may also have been the strategy all along – in many cases to market a platform you have to first demonstrate to the market how the platform can be expressed in an application.  Jason Rosenthal seems to acknowledge as much in a recent interview published in the San Francisco Chronicle’s SF Gate blog in August 2013 (prior to their most recent round being publicly announced) where is quoted as saying that the long term Lytro vision is to become  “the new software and hardware stack for everything with a lens and sensor. That’s still cameras, video cameras, medical and industrial imaging, smartphones, the entire imaging ecosystem.”  Jonathan Heiliger, a general partner at North Bridge Venture Partners, in his quote supporting their participation in the latest $40M round supports that vision – [t]he fun you experience when using a Lytro camera comes from the ability to engage with your photos in ways you never could before.  But powering that interactivity is some great software and hardware technology that can be used for everything with a lens and a sensor.”

I am of course intrigued by the suggestion from Rosenthal that Lytro could be developing solutions useful for medical and industrial imaging.  If you are Pelican Imaging, you are of course focusing on the comments relating to “smartphones.”

Pelican Imaging

Pelican Imaging

Image from Pelican Imaging

Pelican Imaging (@pelicanimaging) was founded in 2008 and its current investors include Qualcomm (@Qualcomm), Nokia Growth Partners, Globespan Capital Partners (@Globespancap), Granite Ventures (@GraniteVentures), InterWest Partners (@InterwestVC) and IQT.  Pelican Imaging has raised more than $37M since inception and recently extended its Series C round by adding an investment (undisclosed amount) from Panasonic in August 2013.   Interesting to me is of course the large number of handset manufacturers who have participated in earlier funding rounds, as well as early investment support from In-Q-Tel (IQT), an investment arm aligned with the United States Central Intelligence Agency.

Pelican Imaging has been pretty quiet from a marketing perspective until recently, but no doubt with their recent additional investment from Panasonic and other hardware manufacturers they are making a push to become the embedded plenoptic sensor platform.


Raytrix is a German developer of plenoptic cameras, and has been doing so since 2009.  They have, up until now, primarily focused on using this technology for a host of industrial imaging solutions.   They offer a range of complete plenoptic camera solutions.  A detailed presentation explaining their solutions can also be found on their site and a very interesting video demonstration of the possibilities of a plenoptic video approach for creating 3D videos can be found hosted at the NVIDIA GPU Technology Conference website.  Raytrix has posted a nice example of how they created a depth map and 3D reconstruction using their camera here.   Raytrix plenotpic video cameras can be used for particle image velocimetry (PIV), a method of measuring velocity fields in fluids by tracking how particles move across time.  Raytrix has a video demonstrating these capabilities here.

The Future

For 2014, I believe we will see the following macro-level trends develop in the 3D capture space (these were originally published here).

  • Expansion of light field cameras – Continued acceleration in 3D model and scene reconstruction (both small and large scale using depth sense and time of flight cameras but with an expansion into light field cameras (i.e. like Lytro, Pelican Imaging, Raytrix, as proposed by Apple, etc).
  • Deprecation of 3D capture hardware in lieu of solutions – We will see many companies which had been focusing mostly on data capture pivot more towards a vertical applications stack, deprecating the 3D capture hardware (as it becomes more and more ubiquitous – i.e. plenoptic cameras combined with RTK GPS accurate smartphones).
  • More contraction in the field due to M&A – Continued contraction of players in the capture/modify/make ecosystem, with established players in the commercial 3D printing and scanning market moving into the consumer space (e.g. Stratasys acquiring Makerbot, getting both a 3D scanner and a huge consumer ecosystem with Thingiverse) and with both ends of the market collapsing in to offer more complete solutions from capture to print (e.g. 3D printing companies buying 3D scanner hardware and software companies, vice versa, etc.)
  • Growing open source software alternatives – Redoubled effort on community sourced 3D reconstruction libraries and application software (e.g. Point Cloud Libraries and Meshlab), with perhaps even an attempt made to commercialize these offerings (like the Red Hat model).
  • 3D Sensors everywhere – Starting in 2014, but really accelerating in the years that follow, 3D sensors everywhere (phones, augmented reality glasses, in our cars) which will constantly capture, record and report depth data – the beginnings of a crowd sourced 3D world model.

Over time, I believe that light field cameras will grow to have a significant place in the consumer acquisition of 3D scene information via mobile devices.  They have the benefit of having a relatively small form factor, are a passive imaging system, and can be used in a workflow which consumers already know and understand.   They are of course not a panacea, and ultimately currently suffer similar limitations as does photogrammetry and stereo reconstruction when targets are not used (e.g. difficulty in accurately computing depth data in scenes without a lot of texture, accuracy dependent on depth of the scene from the camera, etc.) but novel approaches to extract more information from a 4D light field hold promise for capturing more accurate 3D depth data from light field cameras.

For consumers, and consumer applications driven from mobile, I predict that light field technologies will take a significant share of sensor technologies, where accuracy is a secondary consideration (at best) and the ease of use, form factor and the “eye candy” quality of the results are most compelling.   Active imaging systems, like those which Apple acquired from PrimeSense certainly have a strong place in the consumer acquisition of 3D data, but in mobile their usefulness maybe limited by the very nature of the sensing technology (e.g. relatively large power draw and form factor, sensor confusion in the presence of multiple other active devices, etc.).


Disruptive Trends in the Content to Make Ecosystem

Back in April I prepared a presentation covering various aspects of the capture/modify/make ecosystem — covering what I thought were (and were going to be) the disruptive forces that would impact 3D scanning and 3D printing over the coming months and years.

I outlined the following disruptive trends:

  • Democratization of low cost 3D capture devices and solutions
  • Commoditization of high accuracy 3D capture devices
  • Democratization of 3D printing along with the Makers movement
  • “Gamified” content capture, creation and modification tools
  • Leveraging crowd sourced design and open source 3D content communities
  • Accelerating investment in 3D capture and creation technologies
  • New processing and interaction paradigms
  • Overarching policy issues

Disruptive Trends

I’ve posted the full presentation (minus embedded videos, sorry!) if you have interest.  Disruptive Trends in Capture to Make (4.5.13).  These trends continue to evolve and hold true – I intend to update these trends with new representative examples which have popped up in the last half of 2013.

Apple Buys Tech Behind Microsoft Kinect (PrimeSense) – 3D Scanning Impact?

[Update: Apple has confirmed the acquisition of PrimeSense for roughly $350M, when originally published the acquisition was still only rumored.]

It has been reported that Apple (@Apple) has acquired PrimeSense (@GoPrimeSense) for $345M.

I have been long on PrimeSense’s depth sensing cameras for a while – I started following them in the months leading up to the original launch of the Microsoft Kinect in the “Project Natal” days (late 2009).  Photogrammetry was always interesting to me as an approach to create 3D models – but the reconstructions tended to fail frequently (and without warning) and always required post-processing.

My interest in PrimeSense technology was primarily twofold: (1) to find a way to leverage the installed base of Microsoft Kinect devices as 3D capture devices (as well as the Xbox Live payment infrastructure) and (2) to build an inexpensive stand-alone 3D scanner based on PrimeSense technology.  I was only more interested after Microsoft published their real-time scene reconstruction research known as KinectFusion.  Hacks like the Harvard “Drill of Depth”s (a Kinect made mobile by attaching it to a battery powered drill, screen and software, circa early 2011) only further piqued my interest about the possibilities.

Drill of Depth

The writing was on the wall for PrimeSense after Microsoft decided to abandon PrimeSense technology and develop their own depth sensing devices for use with the new Xbox One.  PrimeSense had to transition from a lucrative relationship with one large customer (~30M+ units) to a developer of hardware and firmware solutions seeking broader markets.  The OpenNI initiative (an open source project to develop middleware SDK for 3D sensors which was primarily sponsored by PrimeSense) was an attempt to broaden the potential pool of third party developers who would ultimately build solutions around PrimeSense technologies.

There are many PrimeSense powered 3D scanners in the market today – it will be interesting to see whether this pool expands or contracts after the planned Apple acquisition (e.g. will the direction be inward, focusing the PrimeSense technology to be delivered directly with Apple only devices or will they continue to court third parties developers across all types of hardware and software solutions).   The new PrimeSense Capri form factor already allows for entirely new deployment paradigms for this technology, with one more generation the sensor will have shrunk so much that they can be comfortably embedded directly in phone and tablet devices (but with a trade off in data quality if the sensor shrinks too much).

Here is a quick run-down on a non-exhaustive list of PrimeSense powered 3D scanner hardware technology and vendors (note, this isn’t a profile of the universe of software companies that offer solutions around 3D model and scene reconstruction – as there are many):

Standard Microsoft Kinect – the initial movement for using the PrimeSense technology as a 3D scene reconstruction device came from hacks to the original Microsoft Kinect.  The Kinect was hacked to run independently of the Kinect, and ultimately Microsoft decided to embrace these hacks and develop a standalone Kinect SDK.

Microsoft Kinect for PC – Microsoft began selling a Kinect which would directly interface with Windows devices, it also enabled a “near” mode for the depth camera.

Asus XTION (Pro) – This is an Asus OEM of the PrimeSense technology which provides essentially the same functional specifications as delivered in the Microsoft Kinect (they use the same PrimeSense chipset and reference design).

MatterportMatterport (@Matterport) has raised $10M  since the middle of 2012 to develop a camera system, software and cloud infrastructure for scanning interior spaces.  The camera system itself is built around PrimeSense technologies (along with 2D cameras to capture higher quality images to be referenced to the 3D reconstruction created from the PrimeSense cameras).   Most interesting to me that Matterport counts Red Swan  and Felicis Ventures as investors, both which are also invested into Floored (see below).  A few days ago Forbes profiled the use of the Matterport system, the article is worth a read.

Floored – (@Floored3D), formerly known as Lofty, concentrates primarily on developing software to help visualize interior spaces and is concentrating first on the commercial real estate industry.  Floored has raised at least a little over $1M now, including common investors with Matterport.  For more on the relationship between Matterport and Floored, see this TechCrunch article.  Floored’s CEO is Dave Eisenberg, and he gave a great presentation at the TechCrunch NYC Startup Battlefield in late April 2013 explaining Floored’s value proposition.   Floored is definitely filled with brilliant minds, and obviously a whole lot of computer vision folks who understand how difficult it is to attempt to automatically generate 3D models of interior spaces from scan data (of any quality).  To get a sense of what they are currently thinking about, check out the Floored blog.

Lynx A – This was an offering from a start-up in Austin, Texas known as Lynx Labs (@LynxLabsATX) who launched an early 2013 KickStarter campaign for an “all in one” point and shoot 3D camera.  This device was a sensor, combined with a computing device and software which would allow for the real time capturing and rendering of 3D scenes.  The first round of devices shipped in the middle of September 2013.   I do not know for sure, but my assumption is that this device is PrimeSense powered.

DotProduct (@DotProduct3D) with their DPI-7 scanner.   As with the Lynx A camera, this is a PrimeSense powered device, combined with a Google Nexus, and their scene reconstruction software called Phi.3D.  DotProduct claims 2-4mm accuracy at 1m, achieved through a combination of individual sensor calibration, their software, and rejecting sensors which do not achieve spec.  DotProduct announced in late October 2013, at the Intel Capital Global Summit, that Intel Capital (@IntelCapital) had made a seed investment into DotProduct, spearheaded by Intel’s Perceptual Computing Group.

Occipital Structure SensorOccipital (@occipital) is an extremely interesting company based in Boulder and San Francisco, filled with amazing computer vision expertise.  After cutting their teeth on some computer vision applications for generating panoramas on Apple devices, they have bridged into a complete hardware and software stack for 3D data capture and model creation.  Occipital counts the Foundry Group (@foundrygroup) as one of its investors (having invested roughly $7M into Occipital in late 2011).   Occipital completed a very successful KickStarter campaign for its Structure Sensor raising nearly $1.3M.

Occipital Structure Sensor

The Structure Sensor is a PrimeSense powered device which is officially supported on later generation Apple iPad devices.  What is compelling is Occipital’s approach to create an entire developer ecosystem around this device – no doubt building on the Skanect (@Skanect) technology they acquired from ManCTL in June of 2013.  Skanect was one of the best third party applications available which had implemented and made available the Microsoft Fusion technology (allowing for real time 3D scene reconstruction from depth cameras).   If it is true, and Apple in fact does buy PrimeSense, then that is potentially problematic for Occipital’s current development direction if Apple has aspirations for embedding this technology in mobile devices (as opposed to Apple TV).  Even if Apple did want to embed in their iDevices, it would seem then that Occipital becomes an immediately interesting acquisition target (in one swoop you get hardware, and most importantly the computer vision software expertise).  Given the depth of talent at Occipital, I’m sure things are going to work out just fine.

Sense™ 3D Scanner by 3D Systems – This is the newest 3D scanner entrant in this space (announced a few weeks ago) delivered by 3D Systems (@3dsystemscorp), which acquired my former company, Geomagic.  The Sense uses the new PrimeSense Carmine sensor – a further evolution of the PrimeSense depth camera technology, allowing for greater depth accuracy across more pixels in the field (and ultimately reconstruction quality).  PrimeSense has a case study on the Sense.

What Are Competitive/Replacement Technologies for PrimeSense Depth Sensors?

In my opinion, the closest competitor in the market today to PrimeSense technologies are made by a company called SoftKinetic (@softkinetic) with their line of DepthSense cameras, sensors, middleware and software.


On paper, the functional specifications of these devices stack up well against the PrimeSense reference designs.  Unlike PrimeSense, SoftKinetic sells complete cameras, as well as modules and associated software and middleware.  SoftKinetic uses a time of flight (ToF) approach to capture depth data (which is different than PrimeSense).  Softkinetic has provided software middleware to Sony for the PS4 providing a middleware layer for third party developers to create gesture tracking applications using the PlayStation(R)Camera for PS4.   Softkinetic announced a similar middleware deal with Intel to accelerate perceptual computing in the early summer of 2013 too.

There are other companies in the industrial imaging space (who presently develop machine vision cameras or other time of flight scanners) which could provide consumer devices if they chose to (e.g. such as PMD Technologies in Germany).

I believe the true replacement technology, at least in the consumer space, for 3D data acquisition and reconstruction will come from light field cameras as a class in order to provide range data (e.g. z depth), and not necessarily from active imaging solutions.  See my thoughts on this below.

Predictions for 2014 and Beyond

Early in 2013, when I was asked by my friends at Develop3D to predict what 2013 would bring, I said:

In 2013 we will move through the tipping point of the create/modify/make ecosystem.

Low cost 3D content acquisition, combined with simple, powerful tools will create the 3D content pipeline required for more mainstream 3D printing adoption.  

Sensors, like the Microsoft Kinect, the LeapMotion device, and [Geomagic, now 3D Systems’] Sensable haptic devices, will unlock new interaction paradigms with reality, once digitized.  

Despite the innovation, intellectual property concerns will abound, as we are at the dawn of the next ‘Napster’ era, this one for 3D content.

I believe much of that prediction has come/is coming true.

For 2014 I believe we will see the following macro-level trends in the 3D capture space:

  • Expansion of light field cameras – Continued acceleration in 3D model and scene reconstruction (both small and large scale using depth sense and time of flight cameras but with an expansion into light field cameras (i.e. like Lytro (@Lytro) and Pelican Imaging (@pelicanimaging)).
  • Deprecation of 3D capture hardware in lieu of solutions – We will see many companies which had been focusing mostly on data capture pivot more towards a vertical applications stack, deprecating the 3D capture hardware (as it becomes more and more ubiquitous).
  • More contraction in the field due to M&A – Continued contraction of players in the capture/modify/make ecosystem, with established players in the commercial 3D printing and scanning market moving into the consumer space (e.g. Stratasys acquiring Makerbot, getting both a 3D scanner and a huge consumer ecosystem with Thingiverse) and with both ends of the market collapsing in to offer more complete solutions from capture to print (e.g. 3D printing companies buying 3D scanner hardware and software companies, vice versa, etc.)
  • Growing open source alternatives – Redoubled effort on community sourced 3D reconstruction libraries and application software (e.g. Point Cloud Libraries and Meshlab), with perhaps even an attempt made to commercialize these offerings (like the Red Hat model).
  • 3D Sensors everywhere – Starting in 2014, but really accelerating in the years that follow, 3D sensors everywhere (phones, augmented reality glasses, in our cars) which will constantly capture, record and report depth data – the beginnings of a crowd sourced 3D world model.

The Use of Light Field Cameras and 3D Data Acquisition and Reconstruction Will Explode

While the use of light field cameras to create 3D reconstructions are just at their infancy, just like the PrimeSense technology (which was designed to be used for an interaction paradigm, not for capturing depth data), I can see (no pun intended) this one coming.  Light field cameras have a strong benefit of being a passive approach to 3D data acquisition (like photogrammetry).  For what is possible in depth map creation from these types of camera systems, check out this marketing video from Pelican Imaging (note the 3D Systems Cube 3D printer)] and a more technical one here .

Pelican Imaging Sensor

Image from Pelican Imaging.

I will have a separate post looking in more depth at light field cameras as a class including Lytro’s recent new $40M round of funding and the addition of North Bridge.  I believe, after refinement, that they ultimately become a strong solution for consumer mobile devices for 3D content capture because of their size, power needs, passive approach, etc.  In the interim, if you have interest in this space you should read the Pelican Imaging presentation recently made at SIGGRAPH Asia on the PiCam and reproduced in full at the Pelican Imaging site.  Fast forward to pages 10-12 in this technical presentation for an example of using the Pelican Imaging camera to produce a depth map which is then surfaced.

What could ultimately game changing is if we find updated and refined depth sense technology embedded and delivered directly with the next series of smartphones and augmented reality devices (e.g. Google Glass).  In that world, everyone has a 3D depth sensor, everyone is capturing data, and the potentials are limitless for applications which can harvest and act upon that data once captured.

Let the era of crowd sourced world 3D data capture begin!

(. . . but wait, who owns that 3D world database once created. . .

This article was original published on DEVELOP3D on November 18th, 2013, it has been modified since that original posting.

The call for a harmonized “Community License” for 3D Content – Part I.

Ponoko, Cubify, 3DWarehouse, oh my!

Let’s start first by reviewing one of the older services available to host 3D content and see how they address the issue. I am assuming for purposes of this review that all of the terms of service are in fact enforceable (which might be a big assumption!).

Google 3D Warehouse

Google has been publicly hosting and re-distributing 3D content (primarily in their .SKP file format) for a long time by Internet standards, since at least 2006.   Have you ever thought to read the Google terms applicable to the 3DWarehouse service?  If not, you can find them at:

Under Section 5.7.3, you agree not to upload any content which violates the IP of a third party.  That part is not necessarily surprising.  Google wants you (as the user) to be responsible for the content you post.

What might be surprising is that if you do decide to upload content to the service that under Section 11.1(b) you grant Google “the perpetual, sublicensable, irrevocable, worldwide, royalty-free, and non-exclusive license to reproduce, adapt, modify, translate, publish, publicly perform, publicly display and distribute any [c]ontent or derivative works thereof which you submit, post or display on or through, the [s]ervices.”  This license grant is of the typical type associated with works protected by copyright (but not necessarily other IP types). By virtue of Section 11.2, you grant Google the right to extend this right to other companies or organizations which use the services, and under Section 11.1(c), you grant these rights to other users of the services.   Under Section 11.6 you warrant that you have the right to grant the licenses in this section.

While you are not restricted from providing the content you upload to the service to others under different terms (see Section 11.3(a)) even if you remove the content from the service and terminate your relationship with Google, the license granted in Section 11 continues.

OK, as a practical matter, what does this mean?

It means that once you upload content, you have granted virtually an unlimited right for Google, and third parties, to distribute your work, as well as to create derivative works, without any clear attribution rights back to you.  Google does not provide a selective licensing mechanism to allow people who upload content to pick how much (or how little) of their rights they want to grant or extend to downstream users.  If it turns out that you uploaded something that you didn’t own the rights to, by operation of Section 5.7.3 and 11.6 you could be found to be in breach of contract with Google (as well as subject to an independent infringement action from the actual rights holder).


Ponoko has clearly thought through the implications of IP on the capture or create/modify/make ecosystem.  They walk contributors through a list of Creative Commons license types (for an example see here depending on how the user chooses to sell products or product plans.  The Ponoko terms of use can be found here – and a statement on intellectual property here

Unlike the Google approach, Ponoko makes it clear that they do not obtain any underlying rights to objects which are uploaded to their service other than a right to display them to market the objects for sale.  Even this right expires six (6) months after the item has first been uploaded.


Thingiverse  is the community hosting site for 3D printable content run by Makerbot Industries.  The terms of service can be found here –   Thingiverse follows the same general scheme as Ponoko.  In addition to the limited right granted to Thingiverse necessary to incorporate contributed content into the site and services (found in Section 3.2), the contributor is asked to select which Creative Commons license type they want to share their content under, see Section 3.3.  Thingiverse’s DMCA take-down notice can be found here –


GrabCAD bills itself as a community site offering 3D models for mechanical engineers (and roughly 29K models in early March 2012).  GrabCAD’s terms of service can be found here:

GrabCAD generally follows the Google model, and places the burden on IP clearance issues and concerns with the content contributor, see Section 2 “Responsibility of Contributors.”  GrabCAD does include a DMCA notice as well at Section 7.


Shapeways provides a shop to print existing uploaded 3D content, a pathway for “advanced” users to upload their own models for production, as well as a pathway for “Easy Creators” which allows for the simplified personalization of stock content (suck as cuff links, customized sake set, etc.), see:

Shapeways’ terms and conditions, last updated in February 2012, can be found here –  See the subsections titled “user generated content” and “intellectual property rights in the 3D design”.  Except for 3D Designs, if you upload content to the Shapeways site you grant Shapeways a royalty-free right to use, including create derivative works, based on the uploaded content.  By uploading it, the contributor represents that the content does not infringe any other IP rights. This is consistent with the Google approach.

With respect to uploaded 3D designs, the contributor grants Shapeways the limited right to use the design to manufacture the model to fulfill legitimate orders as well as to display it on the website.  It is possible to change the rights allocation so that the model is only displayed on the website (but cannot be produced).  Shapeways approach to uploaded 3D designs is similar to that of Ponoko.

Cubify (from 3D Systems)

Cubify is in the process of launching – they don’t expect to launch until later in 2012.  Their current terms of service can be found and

Cubify will allow for the download of hosted models under several license types, see Section 5 of the Terms of Service, specifically Sections 5(d), (e), and (f).  These are a “standard royalty free license”, and “editorial license only” and “royalty free license with model release.”  To be able to become a contributor, a user must become a “Cubify Artist”, see:

iMaterialize (from Materialise) 

iMaterialize is the service run by Materialise with the primary focus of providing a location for designers (mostly well known, rather than community developed) to host unique, 3D printable objects.  The models themselves are not downloadable so that derivative works can be made from them.  The FAQ is here: the legal terms are found here:  Whether this site develops over time to be more like Ponoko or Thingiverse remains to be seen.

The FAQ, in refreshing non-legalese, states that: “The person who placed the initial order is considered the owner of the model and has the IP rights for that model. By accepting the vendor terms and conditions, you give i.materialise the right to produce copies of the model in return for a designer’s fee. Normally, the owner is also the designer. That is why we call it a designer’s fee. If you didn’t design the model yourself, but want a special designer’s page for the person who did, please contact us for permission to set up a designer page for this designer.”

The iMaterialize website has a very clean UI in order for a designer to set-up a virtual shop, see:

Summary of Approaches

Service License Type Other notable
Google 3Dwarehouse Unique Might be surprising to learn that once uploaded you grant Google a right to share and even commercialize with others.
Ponoko CCL Elegant drop down license “picker” as part of upload process.
Cubify Unique New site and service, full terms likely under development.  Unclear why 3D Systems did not gravitate towards the CCL scheme rather than developing their own nomenclature.
Shapeways Unique Shapeways, in its terms and conditions, makes explicit distinctions between user uploaded content which is a 3D model versus that which is not.
GrabCAD Unique Contributed content is intended for “non-commercial” use unless otherwise specified by the contributor.  Unclear specifically then what an engineer who downloads content is allowed to do with it.   See Section 4.
Thingiverse CCL Thingiverse expressly states that they will share the CCL selected by the contributor with other sites, but that they are not responsible of that secondary site or service fails to adhere to those terms, see the last sentence of Section 3.2
iMaterialise Unique Not a hosting and community site per se.  Intended to provide a commercialization avenue for unique creations from designers.  Very interesting material selector pages –  Geared towards creating a physical model, not facilitating the sharing and modification of models for downstream uses.

Various Approaches and Interests

As you can see from our view from the trenches, vendors are approaching IP concerns in the create-capture/modify/make ecosystem differently.

This is an area that the folks at Creative Commons are clearly watching (the folks behind the CCL), see the post titled “CC and 3D Printing Community.”

The challenge with the current CC licensing schemes is that they were never intended to cover functional content (that which might be covered by IP rights other than copyright).    As the blog above notes –

With the exception of CC0, the Creative Commons licenses are only for granting permissions to use non-software works. The worlds of software and engineering have additional concerns outside of the scope of what is addressed by the CC licenses. 3D printing is a new medium which encompasses both the creative domains of culture and engineering, and often 3D printed works do not fall neatly into either category.

So what does this mean?  Well, basically, that despite best intentions, services that have tried to create a structured approach to IP issues in the 3D printing community by relying on the CCL scheme have done so prematurely.  Creative Commons does not apparently believe that the CCL scheme, as currently drafted, “works” here.  Uh oh and oh no.

What is Needed?

An integrated, harmonized approach to dealing with all IP considerations in the create-capture/modify/make ecosystem is required.

This is a topic which is already being actively examined for remedy in the CCL 4.0 release cycle.  Far beyond me (as there are domain and policy experts considering the interplay of these various schemes) to make a specific, actionable, proposal – other than to highlight that one is in fact desperately needed.  One that the current stakeholders can work together to formulate and adopt from a policy perspective – and then implement as close to uniformly as possible in their offerings.

The answer may come in the form of a new CCL type (but as the above discussion highlights, there are some distinct incompatibilities between the CCL scheme entirely and combined functional and creative works).  The answer may also come in the form of a new rights scheme applicable broadly to this type of content.   What should not acceptable to anyone is where we currently are – a distinct, explicit shade of grey.

This blog was originally published on March 22nd, 2012.

MapBox, Geo Software Platform, Maps $10M from Foundry Group

It is great to see continuing venture capital and public market interest in areas such as data acquisition, unmanned aerial systems, manufacturing, AEC and GIS solutions providers.

MapBox (@MapBox) announced yesterday that it had taken a Series A investment of $10M from Foundry Group (@FoundryGroup).  After three years of bootstrapping the MapBox business, in the words of Eric Gundersen (@ericg), funding lets us plan for years of building the future of geo software, from the ground up.

MapBox is a cloud-based platform which allows for developers to embed geo rich content into their web and mobile offerings.  MapBox sources its mapping data from OpenStreetMap, keeping its operating costs low and without a tie to proprietary back end mapping databases.   It will be interesting to see how MapBox navigates the GIS/Geo Software playing field over the coming years – but more developer choices, relying on crowd-sourced mapping data, could be quite transformational indeed.

Foundry Group continues its string of investments in the technical solutions space.  They were part of a team which invested $30M into Chris Anderson’s (@chr1sa) unmanned aerial systems company 3D Robotics (@3DRobotics) a few weeks ago, which I blogged about here and were also invested into Makerbot (@Makerbot), which was recently acquired by the 3D printing company Stratasys (@Stratasys) (in mid-August 2013) for $403M (+up to $201M in earn-outs).  Seth Levine (@sether) explained some of Foundry Group’s rationale for the MapBox investment here.

Foundry Group is currently also invested into Occipital (@Occipital) which has recently developed a 3D capture device which connects to an iPad, called the Structure Sensor.  Occipital currently has a KickStarter campaign going for the Structure Sensor, and as of today they are only a few thousand dollars shy of the $1M mark. In June 2013 Occipital acquired ManCTL, adding a strong team to an already deep computer vision bench, but in this case on that had the chops to do real time 3D scene reconstruction from PrimeSense powered (a/k/a the Microsoft Kinect) devices.  Foundry Group put $8M into Occipital in August of 2011.

I am very excited to ultimately see what comes from both MapBox and Occipital!

It will be interesting to see whether/if Andreessen Horowitz (@a16z) looks for a big data, geo centric sector investment as well.