Robotic Search in Clutter

I adapted a FarmBot Genesis V1.6 into a Cartesian manipulation platform. The aim was a repairable, configurable workspace that could run the push-and-grasp experiments normally carried out with a more complex robot arm.

I integrated the motion system, end effector, sensing, and host-side learning pipeline. The FarmDuino handled motor control and power distribution, the Raspberry Pi ran FarmBot OS, and NVIDIA GPU hardware handled the neural-network computation.

Before scaling up the experiments, I checked the motion budget. Figure 6 compares translation times under default and boosted settings: the time spent moving matters when an experiment needs hundreds or thousands of interactions.

An external, angled camera made the observation pipeline sensitive to calibration changes and missing depth data. I moved the Intel RealSense RGB-D camera to a wrist-mounted, top-down arrangement.

That gave the camera a known relationship to the end effector. Starting from the robot position and the camera offset, the pipeline could relate a pixel in the image to an action in the workspace.

Figure 7 shows the physical integration. Figure 8 compares the visual input: the engineering goal was a more consistent view of the target, with fewer occlusion-related gaps in the depth data.

The first parallel gripper came from an undergraduate senior-design project. It was too heavy for the magnetic tool mount and had servo-control problems, so I tested lighter articulated designs.

The eventual PETG gripper adapted a design from an online repository for the FarmBot z-axis. Contact surfaces evolved from rubber bands to molded hot-glue tips, improving contact during pushing and grasping.

This was also an electrical integration problem. Servo power faults could damage the printed mechanism, so reliable power delivery and mechanical limits were part of making the tool usable for repeated trials.

A learned action is only useful if the camera and robot agree about where it should happen. I examined calibration, projection and deprojection, and the crop used to turn the camera stream into the network input.

The pipeline worked with a 224 by 224 representation derived from the RGB-D stream. Cropping, object scale, and depth interpretation all affected whether an action selected in the image landed in the right physical place.

The calibration photograph and the paired simulated/physical views show this part of the work. Improving that consistency enabled repeatable observations; sustained real-world online training remained outside the thesis scope.

I compared segmentation approaches for the controlled target-and-block setup. Lightweight RefineNet struggled with similar colors and changing conditions; K-means offered a practical way to isolate the visible target in these experiments.

The yellow region in Figure 12 is the target mask: pixels the system treats as belonging to the object it wants. This is a cue for deciding where to act, and later becomes a measure of progress when more of the target is uncovered.

I then reproduced the earlier Grasping the Invisible (GTI) pipeline on the physical platform using basic, moderate, and complex clutter. Figures 13 and 14 connect the test arrangements to the observed number of moves. GTI supplied the prior search-and-grasp framework; my work integrated it with this platform and investigated its limitations.

Physical testing exposed two problems: camera distance changed the scale of the input, and a tiny visible fragment could be treated as enough evidence to move into grasp coordination before the target was fully uncovered.

I also investigated searching across four workspace regions. The early heuristic compared normalized action values between the regions to choose where to push next. Its behavior was inconsistent, so I deferred it rather than treating it as a solved scaling method.

These tests motivated a more focused question: could I explicitly train a policy to uncover the object before handing it to the existing singulation and grasping pipeline? A Q-map, shown here, is the network’s estimate of how useful each candidate action will be.

I built a programmatic scene-generation setup around structured clutter: bridge-like blocks and covering blocks arranged around the target. Configuration data specified the meshes, positions, and rotations.

The simulator spawned these arrangements at varied positions and orientations without overlap. This made repeated training and evaluation possible while keeping the task recognizable.

The variation was useful, but it came from a particular family of block arrangements. That matters when interpreting the results: learning these scenes does not establish reliable behavior in arbitrary household clutter.

For this stage I trained the pushing network, leaving the grasping network out of the new training loop. The baseline reward followed the change in visible target-mask area: a push that exposes more of the target provides useful feedback.

When the target is fully hidden, there is no visible target mask to guide the first action. I introduced a pseudo-target region as a search cue. It tells the policy where to investigate; it does not mean the camera has detected the hidden object.

The six variants separate changes to the reward, the spatial cue, and initialization from an existing GTI policy. This makes the comparison more informative than changing every part of the system at once.

I trained each of the six variants for 3,000 iterations. Figures 20 and 21 show the difference between the comparatively stable baseline run and the more oscillatory fine-tuned, pseudo-guided run.

Figure 22 puts the loss traces together. Loss tracks the learning objective, so it helps diagnose training behavior; it is not itself a measure of successful retrieval.

The thesis reports noticeable behavioral improvement around 1,500-2,000 iterations. It also records a weakness shared by the models: performance deteriorated when the structured clutter became highly disordered.

I evaluated the trained policies for 100 iterations each in dynamically generated, stylized clutter. Figure 23 preserves the original mean-and-error-margin plot, including the original GTI policy as a comparison.

The baseline was the strongest overall policy in the thesis discussion, and the pseudo-guided policy was useful for revealing hidden targets. Fine-tuning sometimes reduced the actions needed, but retained more disruptive behavior toward surrounding clutter.

That distinction is important: taking fewer actions and disturbing less of the scene are different outcomes. Figure 24 shows the baseline’s action preferences across clutter arrangements and candidate push orientations.

I carried out initial physical feasibility tests with the baseline and pseudo policies. In fully occluded scenes, I manually placed the pseudo-target region over the clutter to give the guided policy a starting location.

The resulting strategy separates three jobs: use the pseudo policy to begin revealing a target, use the baseline to uncover more of it, and use the existing GTI pipeline to create grasping room and retrieve it.

These were limited physical tests, not a broad real-world benchmark. The main completed evaluation was in simulation; autonomous target-region selection and sustained online learning were still open work.

The completed work brought together the physical platform, more consistent perception, automatic scene generation, six uncovering-policy experiments, and initial physical validation. The sequence below captures the proposed handoff between specialized policies.

The next research direction was longer autonomous experiments with less manual setup. That would require dependable scene resets, broader clutter evaluation, and a way to supply useful search regions without placing them by hand.

The final chapter also considers RF-based localization, thermal sensing, and transformer-based methods as possible future inputs and models. Those are research directions discussed in the thesis, not capabilities implemented or evaluated here.