Category Archives: AEC

Apple’s iPhone X – Bringing PrimeSense 3D Scanning Technology to the Masses

Way back in 2013 (it feels way back given how fast the market continues to move on reality capture hardware and software, AR/VR applications, etc) I blogged about Apple’s acquisition of PrimeSense, and what that meant for the potential future of low cost 3D capture devices.  At the time of the acquisition, PrimeSense technology was being incorporated into a host of low cost (and admittedly relatively low accuracy) 3D capture devices, almost all leveraging the Microsoft Research KinectFusion algorithms developed as against the original Microsoft Kinect (which was based on PrimeSense tech itself).

I, and many others, have wondered when the PrimeSense technology would see the light of day.  After many rumored uses (e.g. use to drive gesture control of Apple TV, as one among many), the PrimeSense tech pipeline has emerged as the core technology behind the 3D face recognition technology which has replaced the fingerprint reader on the iPhone X.  Apple has branded the PrimeSense module as the “TrueDepth” camera.

It would surprise me if there wasn’t work already underway to use the PrimeSense technology in the iPhone X to act as a 3D scanner of objects generally –  as ultimately as enabled by/through the Apple ARKit.  Others, like those at Apple Insider, have come to the same conclusion. As one example, the TrueDepth camera could be used to capture higher quality objects to be placed within the scene that the ARKit can otherwise detect and map to (surfaces, etc.). In another, the TrueDepth camera combined with the data generated from the onboard sensor package combined with known SLAM implementations, and cloud processing, could turn the iPhone X into a mapping and large scene capture device as well as enabling the device to better localize itself within an environment that would be difficult for the device to currently work in (e.g. a relatively featureless space). The challenge with all active sensing technologies (the Apple “TrueDepth” camera, the Intel RealSense camera, or the host of commercial data acquisition devices that are available) is that they are all relatively power hungry, and therefore inefficient as a small form factor, mobile, sensing device (that, oh yeah, needs to be a phone and have long battery life).

Are we at the point where new mobile sensor packages (whether consumer or professional) coupled with new algorithms, fast(er) data transmission and cloud based GPU compute solutions will create the platform to enable crowd sourced world 3D data capture (e.g. Mapillary for the 3D world?). The potential applications working against such a dataset are virtually limitless (and truly exciting!).

Mapillary Raises $8M – Crowdsourced Street Photos

Crowdsourced Street Maps – Mapillary Raises $8M

Mapillary, a Malmo, Switzerland based company that is building a crowdsourced street level photo mapping service has raised $8M in their Series A fundraising round, led by Atomico, with participation by Sequoia Capital, LDV Capital and Playfair Capital.   Some have commented that Mapillary wants to compete with Google Street View using crowd sourced, and then geo-located, photos (and presumably video and other assets over time).  Mapillary uses Mapbox as its base mapping platform.  Mapbox itself sources its underlying street mapping data from OpenStreetMap, as well as satellite imagery, terrain and places information from other commercial sources – you can see the full list here.   Very interesting to see that Mapillary has a relationship with ESRI – such that ESRI ArcGIS users can access Mapillary crowd sourced photo data directly via ArcGIS online.

I previously wrote about MapBox and OpenStreetMap in October 2013 when it closed its initial $10M Series A round led by Foundry Group.  You can see that initial blog post here.  MapBox subsequently raised a $52.6M Series B round, led by DFJ, in June of 2015.  I then examined the intersection of crowdsourced data collection and commercial use in the context of the Crunchbase dispute with Pro Populi and contrasted that with the MapBox and OpenStreetMap relationship.

I am fascinated by the opportunities that are unlocked by the continuing improvement in mobile imaging sensors.  The devices themselves are becoming robust enough for local computer vision processing (rather than sending data to the cloud) and we are perhaps a generation away (IMHO) from having an entirely different class of sensors to capture data from. That combined with significant improvements in location services makes it possible to explore some very interesting business and data services in the future.

In late 2013 I predicted that, in time, mobile 3D capture devices (and primarily passive ones) would ultimately be used to capture, and tie together a crowd sourced world model of 3D data.

What could ultimately game changing is if we find updated and refined depth sense technology embedded and delivered directly with the next series of smartphones and augmented reality devices. . .  In that world, everyone has a 3D depth sensor, everyone is capturing data, and the potentials are limitless for applications which can harvest and act upon that data once captured.

Let the era of crowd sourced world 3D data capture begin!

It makes absolute sense that the place to start along this journey is 2D and video imagery, which can be supplement (and ultimately supplanted) over time by richer sources of data  – leveraging an infrastructure that has already been built.  We still have thorny and interesting intellectual property implications to consider (Think Before You Scan That Building) – but regardless – bravo Mapillary! Bravo indeed!

Think Before You 3D Scan That Building

There is a growing awareness and understanding of the intellectual property considerations in the capture/modify/make ecosystem – particularly as it relates to content that is captured via a 3D scanner, modified or mashed up, and then manufactured (via 3D printing or otherwise).   I have written about this before – blogging early in 2012 in The Storm Clouds on the Horizon that I felt the next “Napster” era was upon us for digitally captured real world content.   In the last two years there have been transformative technical changes on both “ends” of that ecosystem, consumer/prosumer 3D printing solutions along with an emerging class of inexpensive 3D real world capture devices and software solutions.

Earlier in 2014 I identified the following key trends in the capture/modify/make ecosystem for object based 3D capture and reproduction:

2014 Market Trends

Intellectual Property Concerns in Scene Based 3D Capture?

I’d point you to the on-going discussion around the metes and bounds of intellectual property protection in the object based capture/modify/make ecosystem is interesting context for some high level issue spotting with the intersection of intellectual property and scene/world based reality capture (as part of a formal digital documentation process or as part of an informal, crowd-sourced creation of a 3D world model).   What you’ll find is a lot of gray areas, inconsistencies, and mind melting information.

Understanding intellectual property in the context of scene based scanning will become more relevant over the coming years for many of the same reasons that have driven change and awareness in the context of capturing and reproducing objects.

The falling cost of high accuracy scanners useful for scene based 3D capture (e.g. the FARO Focus3D family means that more data will be captured, by more users, in the commercial context (either directly by the owner/operators themselves or by third party scanning service bureaus).  Similarly, 3D data capture will become ubiquitous on the consumer side later in 2014 and beyond – as Intel adds their RealSenseTM depth sense technology to every laptop they ship, Google progresses with Project Tango along with their software partners and as lightfield camera technology goes mainstream.  In my opinion, we are not too far away from the creation and continuous update of a 3D “world model” populated with data coming from various types of consumer, professional and industrial sensors.

Intellectual Property Applies to Buildings and Public Spaces?

I hate to be the bearer of bad news, but to answer the question, “yes” intellectual property rights may be impacted if you capture 3D data of a building, public spaces or other real world scenes. That’s probably surprising to many of you.  For purposes of this discussion I’m only going to focus on how the laws in the United States might apply (and in many respects they differ from other countries around the world – for an example of just how interesting/different that can be, see:  Your mileage can and will vary, this is presented to help with issue spotting – and not to provide definitive guidance on any particular situation.

I had the opportunity to speak on this topic with Michael Weinberg from Public Knowledge at a FARO Digital Documentation Conference a few years ago – many of the issues and examples we discussed then are still relevant now (and even more so with the explosion of low cost 3D capture solutions).

You Can Infringe Intellectual Property By Scanning A [Building] [Lobby] [Plaza] [etc.]?

The intellectual property ramifications of capturing 3D data in the context of a scene is very muddled.  Very little case law addresses these issues, and that which does, isn’t very clear.  Naturally, most buildings would not be copyrightable because, by their very nature, they are “useful” – and something that is useful is generally not afforded copyright protection (yes, useful objects can be patented, and you can actually patent certain building elements – but that’s a topic for a different day).

But think about something like a sculptural memorial that is in a public plaza.  Would the sculptor of the memorial be afforded copyright protection?  You betcha, as a sculptural work is specifically protected under the United States Copyright Act.   What if you were to take a picture of that memorial and decide to license it to the United States Postal Service for use on a postage stamp.  Would that picture, and re-use, require you to obtain clearance (in the form of a license) from the sculptor before you could do so, and before the USPS could sell the stamps?
Korean War Veterans Memorial Sculpture

According to the 5th Circuit Court of Appeals, when they looked at this question in 2010 – the answer was “yes.”  The failure to get a license from the sculptor, even though the defendant obtained one from the general contractor which installed the sculpture, constituted copyright infringement.   After remand back to the United States Court of Federal Claims, and a subsequent appeal, the sculptor, Frank Gaylord, was awarded $685,000 in the fall of 2013.

So what if you plopped your scanner in the middle of that field and captured a 3D point cloud (that mere act is likely infringing)?   What if you decided to sell that data to a third party and then printed 3D prints from the data?  What if you used that data as part of an immersive augmented reality platform to promote tourism for Washington DC?   Would you/could you be liable?

OK Sculptures Maybe, But A Building?

Empire State Building

A building is the essence of utilitarian and functional, so we are safe from copyright, right?  You might think so.  But you’d be wrong.

Cooper Union Building

Take for example the above, which is the Cooper Union New Academic Building in New York City. Construction was finished in 2009.   Still utilitarian and functional (umm, as a building) so not copyrightable, right?

Wrong. In the United States, under the Architectural Works Copyright Act of 1990 a building designed (and that design is fixed in a tangible medium – i.e. drawings, or actually constructed) after 1990 is specifically subject to copyright protection (although purely functional or utilitarian aspects of a building are not protected).  So, if you set up your 3D scanner on the sidewalk here and captured a point cloud of the Cooper Union New Academic Building, have you committed an act of copyright infringement?

Alright how about this awesome bridge –


It’s a bridge, so the essence of utilitarian and functional.  It also has significant artistic and sculptural elements. Since it’s not intended for human habitation, the Architectural Works Copyright Act does not apply.  Phew.

How about this pavilion built at the Fort Maurepaus beach park, located in Ocean Springs, Mississippi and constructed by FEMA after Hurricane Kartrina?


Folks aren’t supposed to live in pavilions, so we are safe, right?  Wrong.  Pavilions are specifically covered by the Architectural Works Copyright Act.  Only if it was built after 1990.  And it was. Oye.

 But Wait, We Have An Exception!

There is an exception to acts of infringement under the Architectural Works Copyright Act –

 The copyright in an architectural work that has been constructed does not include the right to prevent the making, distributing, or public display of pictures, paintings, photographs, or other pictorial representations of the work, if the building in which the work is embodied is located in or ordinarily visible from a public place.

United States Copyright Act, 17 U.S.C. Section 120 (as amended).

Yippee, so if we user our 3D scanner to capture a point cloud of a building we are saved by the so-called “photographer’s exception” for buildings that have been constructed and can be seen from a public place.  Right?  Unfortunately it’s a big “I have no idea but don’t think so but couldn’t find any case law that answers this question”.

Baltimore Train Station

The above is a photo of Baltimore Penn Station.  This was certainly built before 1990 – so no protection under the AWCA.   Even if it was built after 1990 – we would still OK because it would be covered by the photographer’s exception (assuming that applied to 3D data acquisition), right?

Wrong.  Even if that exception were extended to 3D data capture via scanning, that specific statutory exception does not apply to sculptures (or other objects protected by copyright) which are separable from a building.  [FWIW, the sculpture is called Male/Female by Jonathan Borofsky].

Other countries (e.g. Canada, Ireland, the UK) extend the photographer exception concept to all publicly located, but otherwise copyrightable, works (e.g. sculptures).  Not in the United States though.

Think You Are Confused Now?

What about a sculpture in a building?

Sculpture Attached to a Building

Or how about a sculpture that is attached to a building?

Sculpture in a Building

Or what about a building built by Yale in 2005, CALLED the “Sculpture Building” –

The Sculpture Building

Or how about a building that IS a sculpture (like the Walt Disney Concert Hall in Los Angeles, designed by Frank Gehry?)

Frank Gehry Concert Hall

Policy Implications of Ubiquitous 3D Data Capture of Scenes

In addition to the various intellectual property concerns that are potentially touched about when 3D scenes are captured, I believe there a host of other privacy and ownership issues that need to be thought through as well.  If I’m a facility owner, and I don’t want data captured – how do I prevent it as devices become more ubiquitous?  Sure, I can require people to leave their phones at the security desk (many secure facilities already have no photography or data transfer processes), but what do I do about their glasses?   If I’m a contributor of 3D data to a community sourced 3D “world model” who owns the data that I capture and upload?   Who is responsible if it is ultimately found to be infringing?   What are the policy and legal implications if Google, instead of capturing photographs for their street maps, instead, created 3D point clouds of every place they went?

Some Practical Advice for Service Providers

So what can you do minimize your risks if you are a commercial scanning service provider and are engaged to do some scene based scanning?

  • Ask questions —  Know enough to generally understand the potential risks and pitfalls of any data capture engagement.
  • Transfer liability and responsibility for clearance  – As a service provider, make sure that the owner/operator or the entity which engaged you to do complete the work is responsible for intellectual property clearance issues and agree to hold you harmless (e.g. they are responsible, not you, for any potential infringements).
  • Be especially careful with artistic elements –  Creative and sculptural elements should be subject to more scrutiny.  For example, if you are asked to scan a building lobby, and there is a sculpture in the middle of it, you should specifically get clearance from the artist.
  • Know how the collected data will be used —  Be absolutely clear on the data ownership and the plans for downstream use.  Is the data going to be used as part of a digital documentation process (so no broad public dissemination) or is going to be published and made accessible as part of an augmented reality application?

About the Author

Tom Kurke was the former President and Chief Operating Officer of Geomagic, a specialist supplier of 3D reconstruction and interaction software and hardware solutions – which was acquired by 3D Systems Corporation (NYSE: DDD) earlier this year.  Prior to Geomagic he spent more than a decade with Bentley Systems, a leading providing of solutions to the designers, constructors owners and operators of some of the largest constructed assets in the world.  He recently joined the Board of Advisors of Paracosm ( whose mission is to “3D-ify the World.”  When not supporting his two sons various sporting activities, or writing on topics of interest in the areas of 3D printing, digital reality capture, intellectual property, AEC/GIS or unmanned aerial systems at, you might see him finding new ways to crash his quadcopter.

[Note: This article was originally published on LiDAR News on April 26th, 2014, you can find that here.]